Simplify the code

Conflicts:
	python/obitools3/obidms/_obidmscolumn_seq.pyx

LICENSE

@@ -0,0 +1,518 @@
+
+  CeCILL FREE SOFTWARE LICENSE AGREEMENT
+
+Version 2.1 dated 2013-06-21
+
+
+    Notice
+
+This Agreement is a Free Software license agreement that is the result
+of discussions between its authors in order to ensure compliance with
+the two main principles guiding its drafting:
+
+  * firstly, compliance with the principles governing the distribution
+    of Free Software: access to source code, broad rights granted to users,
+  * secondly, the election of a governing law, French law, with which it
+    is conformant, both as regards the law of torts and intellectual
+    property law, and the protection that it offers to both authors and
+    holders of the economic rights over software.
+
+The authors of the CeCILL (for Ce[a] C[nrs] I[nria] L[ogiciel] L[ibre]) 
+license are: 
+
+Commissariat à l'énergie atomique et aux énergies alternatives - CEA, a
+public scientific, technical and industrial research establishment,
+having its principal place of business at 25 rue Leblanc, immeuble Le
+Ponant D, 75015 Paris, France.
+
+Centre National de la Recherche Scientifique - CNRS, a public scientific
+and technological establishment, having its principal place of business
+at 3 rue Michel-Ange, 75794 Paris cedex 16, France.
+
+Institut National de Recherche en Informatique et en Automatique -
+Inria, a public scientific and technological establishment, having its
+principal place of business at Domaine de Voluceau, Rocquencourt, BP
+105, 78153 Le Chesnay cedex, France.
+
+
+    Preamble
+
+The purpose of this Free Software license agreement is to grant users
+the right to modify and redistribute the software governed by this
+license within the framework of an open source distribution model.
+
+The exercising of this right is conditional upon certain obligations for
+users so as to preserve this status for all subsequent redistributions.
+
+In consideration of access to the source code and the rights to copy,
+modify and redistribute granted by the license, users are provided only
+with a limited warranty and the software's author, the holder of the
+economic rights, and the successive licensors only have limited liability.
+
+In this respect, the risks associated with loading, using, modifying
+and/or developing or reproducing the software by the user are brought to
+the user's attention, given its Free Software status, which may make it
+complicated to use, with the result that its use is reserved for
+developers and experienced professionals having in-depth computer
+knowledge. Users are therefore encouraged to load and test the
+suitability of the software as regards their requirements in conditions
+enabling the security of their systems and/or data to be ensured and,
+more generally, to use and operate it in the same conditions of
+security. This Agreement may be freely reproduced and published,
+provided it is not altered, and that no provisions are either added or
+removed herefrom.
+
+This Agreement may apply to any or all software for which the holder of
+the economic rights decides to submit the use thereof to its provisions.
+
+Frequently asked questions can be found on the official website of the
+CeCILL licenses family (http://www.cecill.info/index.en.html) for any 
+necessary clarification.
+
+
+    Article 1 - DEFINITIONS
+
+For the purpose of this Agreement, when the following expressions
+commence with a capital letter, they shall have the following meaning:
+
+Agreement: means this license agreement, and its possible subsequent
+versions and annexes.
+
+Software: means the software in its Object Code and/or Source Code form
+and, where applicable, its documentation, "as is" when the Licensee
+accepts the Agreement.
+
+Initial Software: means the Software in its Source Code and possibly its
+Object Code form and, where applicable, its documentation, "as is" when
+it is first distributed under the terms and conditions of the Agreement.
+
+Modified Software: means the Software modified by at least one
+Contribution.
+
+Source Code: means all the Software's instructions and program lines to
+which access is required so as to modify the Software.
+
+Object Code: means the binary files originating from the compilation of
+the Source Code.
+
+Holder: means the holder(s) of the economic rights over the Initial
+Software.
+
+Licensee: means the Software user(s) having accepted the Agreement.
+
+Contributor: means a Licensee having made at least one Contribution.
+
+Licensor: means the Holder, or any other individual or legal entity, who
+distributes the Software under the Agreement.
+
+Contribution: means any or all modifications, corrections, translations,
+adaptations and/or new functions integrated into the Software by any or
+all Contributors, as well as any or all Internal Modules.
+
+Module: means a set of sources files including their documentation that
+enables supplementary functions or services in addition to those offered
+by the Software.
+
+External Module: means any or all Modules, not derived from the
+Software, so that this Module and the Software run in separate address
+spaces, with one calling the other when they are run.
+
+Internal Module: means any or all Module, connected to the Software so
+that they both execute in the same address space.
+
+GNU GPL: means the GNU General Public License version 2 or any
+subsequent version, as published by the Free Software Foundation Inc.
+
+GNU Affero GPL: means the GNU Affero General Public License version 3 or
+any subsequent version, as published by the Free Software Foundation Inc.
+
+EUPL: means the European Union Public License version 1.1 or any
+subsequent version, as published by the European Commission.
+
+Parties: mean both the Licensee and the Licensor.
+
+These expressions may be used both in singular and plural form.
+
+
+    Article 2 - PURPOSE
+
+The purpose of the Agreement is the grant by the Licensor to the
+Licensee of a non-exclusive, transferable and worldwide license for the
+Software as set forth in Article 5 <#scope> hereinafter for the whole
+term of the protection granted by the rights over said Software.
+
+
+    Article 3 - ACCEPTANCE
+
+3.1 The Licensee shall be deemed as having accepted the terms and
+conditions of this Agreement upon the occurrence of the first of the
+following events:
+
+  * (i) loading the Software by any or all means, notably, by
+    downloading from a remote server, or by loading from a physical medium;
+  * (ii) the first time the Licensee exercises any of the rights granted
+    hereunder.
+
+3.2 One copy of the Agreement, containing a notice relating to the
+characteristics of the Software, to the limited warranty, and to the
+fact that its use is restricted to experienced users has been provided
+to the Licensee prior to its acceptance as set forth in Article 3.1
+<#accepting> hereinabove, and the Licensee hereby acknowledges that it
+has read and understood it.
+
+
+    Article 4 - EFFECTIVE DATE AND TERM
+
+
+      4.1 EFFECTIVE DATE
+
+The Agreement shall become effective on the date when it is accepted by
+the Licensee as set forth in Article 3.1 <#accepting>.
+
+
+      4.2 TERM
+
+The Agreement shall remain in force for the entire legal term of
+protection of the economic rights over the Software.
+
+
+    Article 5 - SCOPE OF RIGHTS GRANTED
+
+The Licensor hereby grants to the Licensee, who accepts, the following
+rights over the Software for any or all use, and for the term of the
+Agreement, on the basis of the terms and conditions set forth hereinafter.
+
+Besides, if the Licensor owns or comes to own one or more patents
+protecting all or part of the functions of the Software or of its
+components, the Licensor undertakes not to enforce the rights granted by
+these patents against successive Licensees using, exploiting or
+modifying the Software. If these patents are transferred, the Licensor
+undertakes to have the transferees subscribe to the obligations set
+forth in this paragraph.
+
+
+      5.1 RIGHT OF USE
+
+The Licensee is authorized to use the Software, without any limitation
+as to its fields of application, with it being hereinafter specified
+that this comprises:
+
+ 1. permanent or temporary reproduction of all or part of the Software
+    by any or all means and in any or all form.
+
+ 2. loading, displaying, running, or storing the Software on any or all
+    medium.
+
+ 3. entitlement to observe, study or test its operation so as to
+    determine the ideas and principles behind any or all constituent
+    elements of said Software. This shall apply when the Licensee
+    carries out any or all loading, displaying, running, transmission or
+    storage operation as regards the Software, that it is entitled to
+    carry out hereunder.
+
+
+      5.2 ENTITLEMENT TO MAKE CONTRIBUTIONS
+
+The right to make Contributions includes the right to translate, adapt,
+arrange, or make any or all modifications to the Software, and the right
+to reproduce the resulting software.
+
+The Licensee is authorized to make any or all Contributions to the
+Software provided that it includes an explicit notice that it is the
+author of said Contribution and indicates the date of the creation thereof.
+
+
+      5.3 RIGHT OF DISTRIBUTION
+
+In particular, the right of distribution includes the right to publish,
+transmit and communicate the Software to the general public on any or
+all medium, and by any or all means, and the right to market, either in
+consideration of a fee, or free of charge, one or more copies of the
+Software by any means.
+
+The Licensee is further authorized to distribute copies of the modified
+or unmodified Software to third parties according to the terms and
+conditions set forth hereinafter.
+
+
+        5.3.1 DISTRIBUTION OF SOFTWARE WITHOUT MODIFICATION
+
+The Licensee is authorized to distribute true copies of the Software in
+Source Code or Object Code form, provided that said distribution
+complies with all the provisions of the Agreement and is accompanied by:
+
+ 1. a copy of the Agreement,
+
+ 2. a notice relating to the limitation of both the Licensor's warranty
+    and liability as set forth in Articles 8 and 9,
+
+and that, in the event that only the Object Code of the Software is
+redistributed, the Licensee allows effective access to the full Source
+Code of the Software for a period of at least three years from the
+distribution of the Software, it being understood that the additional
+acquisition cost of the Source Code shall not exceed the cost of the
+data transfer.
+
+
+        5.3.2 DISTRIBUTION OF MODIFIED SOFTWARE
+
+When the Licensee makes a Contribution to the Software, the terms and
+conditions for the distribution of the resulting Modified Software
+become subject to all the provisions of this Agreement.
+
+The Licensee is authorized to distribute the Modified Software, in
+source code or object code form, provided that said distribution
+complies with all the provisions of the Agreement and is accompanied by:
+
+ 1. a copy of the Agreement,
+
+ 2. a notice relating to the limitation of both the Licensor's warranty
+    and liability as set forth in Articles 8 and 9,
+
+and, in the event that only the object code of the Modified Software is
+redistributed,
+
+ 3. a note stating the conditions of effective access to the full source
+    code of the Modified Software for a period of at least three years
+    from the distribution of the Modified Software, it being understood
+    that the additional acquisition cost of the source code shall not
+    exceed the cost of the data transfer.
+
+
+        5.3.3 DISTRIBUTION OF EXTERNAL MODULES
+
+When the Licensee has developed an External Module, the terms and
+conditions of this Agreement do not apply to said External Module, that
+may be distributed under a separate license agreement.
+
+
+        5.3.4 COMPATIBILITY WITH OTHER LICENSES
+
+The Licensee can include a code that is subject to the provisions of one
+of the versions of the GNU GPL, GNU Affero GPL and/or EUPL in the
+Modified or unmodified Software, and distribute that entire code under
+the terms of the same version of the GNU GPL, GNU Affero GPL and/or EUPL.
+
+The Licensee can include the Modified or unmodified Software in a code
+that is subject to the provisions of one of the versions of the GNU GPL,
+GNU Affero GPL and/or EUPL and distribute that entire code under the
+terms of the same version of the GNU GPL, GNU Affero GPL and/or EUPL.
+
+
+    Article 6 - INTELLECTUAL PROPERTY
+
+
+      6.1 OVER THE INITIAL SOFTWARE
+
+The Holder owns the economic rights over the Initial Software. Any or
+all use of the Initial Software is subject to compliance with the terms
+and conditions under which the Holder has elected to distribute its work
+and no one shall be entitled to modify the terms and conditions for the
+distribution of said Initial Software.
+
+The Holder undertakes that the Initial Software will remain ruled at
+least by this Agreement, for the duration set forth in Article 4.2 <#term>.
+
+
+      6.2 OVER THE CONTRIBUTIONS
+
+The Licensee who develops a Contribution is the owner of the
+intellectual property rights over this Contribution as defined by
+applicable law.
+
+
+      6.3 OVER THE EXTERNAL MODULES
+
+The Licensee who develops an External Module is the owner of the
+intellectual property rights over this External Module as defined by
+applicable law and is free to choose the type of agreement that shall
+govern its distribution.
+
+
+      6.4 JOINT PROVISIONS
+
+The Licensee expressly undertakes:
+
+ 1. not to remove, or modify, in any manner, the intellectual property
+    notices attached to the Software;
+
+ 2. to reproduce said notices, in an identical manner, in the copies of
+    the Software modified or not.
+
+The Licensee undertakes not to directly or indirectly infringe the
+intellectual property rights on the Software of the Holder and/or
+Contributors, and to take, where applicable, vis-à-vis its staff, any
+and all measures required to ensure respect of said intellectual
+property rights of the Holder and/or Contributors.
+
+
+    Article 7 - RELATED SERVICES
+
+7.1 Under no circumstances shall the Agreement oblige the Licensor to
+provide technical assistance or maintenance services for the Software.
+
+However, the Licensor is entitled to offer this type of services. The
+terms and conditions of such technical assistance, and/or such
+maintenance, shall be set forth in a separate instrument. Only the
+Licensor offering said maintenance and/or technical assistance services
+shall incur liability therefor.
+
+7.2 Similarly, any Licensor is entitled to offer to its licensees, under
+its sole responsibility, a warranty, that shall only be binding upon
+itself, for the redistribution of the Software and/or the Modified
+Software, under terms and conditions that it is free to decide. Said
+warranty, and the financial terms and conditions of its application,
+shall be subject of a separate instrument executed between the Licensor
+and the Licensee.
+
+
+    Article 8 - LIABILITY
+
+8.1 Subject to the provisions of Article 8.2, the Licensee shall be
+entitled to claim compensation for any direct loss it may have suffered
+from the Software as a result of a fault on the part of the relevant
+Licensor, subject to providing evidence thereof.
+
+8.2 The Licensor's liability is limited to the commitments made under
+this Agreement and shall not be incurred as a result of in particular:
+(i) loss due the Licensee's total or partial failure to fulfill its
+obligations, (ii) direct or consequential loss that is suffered by the
+Licensee due to the use or performance of the Software, and (iii) more
+generally, any consequential loss. In particular the Parties expressly
+agree that any or all pecuniary or business loss (i.e. loss of data,
+loss of profits, operating loss, loss of customers or orders,
+opportunity cost, any disturbance to business activities) or any or all
+legal proceedings instituted against the Licensee by a third party,
+shall constitute consequential loss and shall not provide entitlement to
+any or all compensation from the Licensor.
+
+
+    Article 9 - WARRANTY
+
+9.1 The Licensee acknowledges that the scientific and technical
+state-of-the-art when the Software was distributed did not enable all
+possible uses to be tested and verified, nor for the presence of
+possible defects to be detected. In this respect, the Licensee's
+attention has been drawn to the risks associated with loading, using,
+modifying and/or developing and reproducing the Software which are
+reserved for experienced users.
+
+The Licensee shall be responsible for verifying, by any or all means,
+the suitability of the product for its requirements, its good working
+order, and for ensuring that it shall not cause damage to either persons
+or properties.
+
+9.2 The Licensor hereby represents, in good faith, that it is entitled
+to grant all the rights over the Software (including in particular the
+rights set forth in Article 5 <#scope>).
+
+9.3 The Licensee acknowledges that the Software is supplied "as is" by
+the Licensor without any other express or tacit warranty, other than
+that provided for in Article 9.2 <#good-faith> and, in particular,
+without any warranty as to its commercial value, its secured, safe,
+innovative or relevant nature.
+
+Specifically, the Licensor does not warrant that the Software is free
+from any error, that it will operate without interruption, that it will
+be compatible with the Licensee's own equipment and software
+configuration, nor that it will meet the Licensee's requirements.
+
+9.4 The Licensor does not either expressly or tacitly warrant that the
+Software does not infringe any third party intellectual property right
+relating to a patent, software or any other property right. Therefore,
+the Licensor disclaims any and all liability towards the Licensee
+arising out of any or all proceedings for infringement that may be
+instituted in respect of the use, modification and redistribution of the
+Software. Nevertheless, should such proceedings be instituted against
+the Licensee, the Licensor shall provide it with technical and legal
+expertise for its defense. Such technical and legal expertise shall be
+decided on a case-by-case basis between the relevant Licensor and the
+Licensee pursuant to a memorandum of understanding. The Licensor
+disclaims any and all liability as regards the Licensee's use of the
+name of the Software. No warranty is given as regards the existence of
+prior rights over the name of the Software or as regards the existence
+of a trademark.
+
+
+    Article 10 - TERMINATION
+
+10.1 In the event of a breach by the Licensee of its obligations
+hereunder, the Licensor may automatically terminate this Agreement
+thirty (30) days after notice has been sent to the Licensee and has
+remained ineffective.
+
+10.2 A Licensee whose Agreement is terminated shall no longer be
+authorized to use, modify or distribute the Software. However, any
+licenses that it may have granted prior to termination of the Agreement
+shall remain valid subject to their having been granted in compliance
+with the terms and conditions hereof.
+
+
+    Article 11 - MISCELLANEOUS
+
+
+      11.1 EXCUSABLE EVENTS
+
+Neither Party shall be liable for any or all delay, or failure to
+perform the Agreement, that may be attributable to an event of force
+majeure, an act of God or an outside cause, such as defective
+functioning or interruptions of the electricity or telecommunications
+networks, network paralysis following a virus attack, intervention by
+government authorities, natural disasters, water damage, earthquakes,
+fire, explosions, strikes and labor unrest, war, etc.
+
+11.2 Any failure by either Party, on one or more occasions, to invoke
+one or more of the provisions hereof, shall under no circumstances be
+interpreted as being a waiver by the interested Party of its right to
+invoke said provision(s) subsequently.
+
+11.3 The Agreement cancels and replaces any or all previous agreements,
+whether written or oral, between the Parties and having the same
+purpose, and constitutes the entirety of the agreement between said
+Parties concerning said purpose. No supplement or modification to the
+terms and conditions hereof shall be effective as between the Parties
+unless it is made in writing and signed by their duly authorized
+representatives.
+
+11.4 In the event that one or more of the provisions hereof were to
+conflict with a current or future applicable act or legislative text,
+said act or legislative text shall prevail, and the Parties shall make
+the necessary amendments so as to comply with said act or legislative
+text. All other provisions shall remain effective. Similarly, invalidity
+of a provision of the Agreement, for any reason whatsoever, shall not
+cause the Agreement as a whole to be invalid.
+
+
+      11.5 LANGUAGE
+
+The Agreement is drafted in both French and English and both versions
+are deemed authentic.
+
+
+    Article 12 - NEW VERSIONS OF THE AGREEMENT
+
+12.1 Any person is authorized to duplicate and distribute copies of this
+Agreement.
+
+12.2 So as to ensure coherence, the wording of this Agreement is
+protected and may only be modified by the authors of the License, who
+reserve the right to periodically publish updates or new versions of the
+Agreement, each with a separate number. These subsequent versions may
+address new issues encountered by Free Software.
+
+12.3 Any Software distributed under a given version of the Agreement may
+only be subsequently distributed under the same version of the Agreement
+or a subsequent version, subject to the provisions of Article 5.3.4
+<#compatibility>.
+
+
+    Article 13 - GOVERNING LAW AND JURISDICTION
+
+13.1 The Agreement is governed by French law. The Parties agree to
+endeavor to seek an amicable solution to any disagreements or disputes
+that may arise during the performance of the Agreement.
+
+13.2 Failing an amicable solution within two (2) months as from their
+occurrence, and unless emergency proceedings are necessary, the
+disagreements or disputes shall be referred to the Paris Courts having
+jurisdiction, by the more diligent Party.

README.md

@@ -0,0 +1,40 @@
+The `OBITools3`: A package for the management of analyses and data in DNA metabarcoding   
+---------------------------------------------
+
+
+
+DNA metabarcoding offers new perspectives for biodiversity research [1]. This approach of ecosystem studies relies heavily on the use of Next-Generation Sequencing (NGS), and consequently requires the ability to to treat large volumes of data. The `OBITools` package  satisfies this requirement thanks to a set of programs specifically designed for analyzing NGS data in a DNA metabarcoding context [2] - <http://metabarcoding.org/obitools>. Their capacity to filter and edit sequences while taking into account taxonomic annotation helps to setup tailored-made analysis pipelines for a broad range of DNA metabarcoding applications, including biodiversity surveys or diet analyses.   
+
+
+
+**The `OBITools3`.** This new version of the `OBITools` looks to significantly improve the storage efficiency and the data processing speed. To this end, the `OBITools3` rely on an ad hoc database system, inside which all the data that a DNA metabarcoding experiment must consider is stored: the sequences, the metadata (describing for instance the samples), the database containing the reference sequences used for the taxonomic annotation, as well as the taxonomic databases. Besides the gain in efficiency, this new structure allows an easier access to all the data associated with an experiment.   
+
+
+
+**Column-oriented storage.** An analysis pipeline corresponds to a succession of commands, each computing one step of the analysis, and where the result of the command *n* is used by the command *n+1*. DNA metabarcoding data can easily be represented in the form of tables, and each command can be regarded as an operation transforming one or several 'input' tables into one or several 'output' tables, which can be used by the next command. Many of the basic operations in a pipeline copy without modification an important part of the input tables to the result tables, and use for their calculations only a small part of the input data. In the original `OBITools`, those tables are kept in the form of annotated sequence files in the FASTA or FASTQ format. This has two consequences: i) keeping the transitional results of the analysis pipeline means using disk space for an important volume of redundant data, ii) The coding and decoding of informations that are not actually used represent an important part of the treatment process. The new database system used by the `OBITools3` (called DMS for Data Management System) relies on column-oriented storage. The columns are immutable and can be assembled in views representing the data tables. This way, the data not modified by a command in an input table can easily be associated to the result table without duplicating any information ; and the data not used at all by a command can be associated with the result table without being read. This strategy results in a gain in disk space efficiency by limiting data redundancy, as well as a gain in execution time by limiting data reading, writing and conversion operations. Finally, as a mean to optimize data access, each column is stored in a binary file directly mapped in memory for reading and writing operations.   
+
+
+
+**Storage optimization.** DNA metabarcoding data is intrinsically very redundant. For example, the same sequence corresponding to a species will be present several thousand times across all samples. In order to limit the disk space used and make comparison operations more efficient, data in the form of character strings is stored in columns using a complex indexing structure, efficient on millions of values, coupling hash functions, Bloom filters and AVL trees. Finally, DNA sequences are compressed by encoding each nucleotide on two or four bits depending on whether the sequences contain only the four nucleotides (A, C, G, T) or use the IUPAC codes.   
+
+
+
+**Saving the data processing history.** The totality of the informations used by the `OBITools3` is stored in immutable data structures in the DMS. If a command has to modify a column used as input to produce its result, a new version of that column is created, leaving the initial version intact. This storage system enables to keep, at minimal cost, the totality of the transitional results produced by the pipeline. The storage of metadata describing all the operations that have produced a view (a result table) in the DMS makes possible the creation of an oriented hypergraph, where each node corresponds to a view and each arrow to an operation. By retracing the dependency relationships in this hypergraph, it is possible to rebuild *a posteriori* the entirety of the process that has produced a result table.   
+
+
+
+**Tools.** The `OBITools3` offer the same tools as the original `OBITools`. Eventually, new versions of `ecoPrimers` (PCR primer design) [3], `ecoPCR` (*in silico* PCR) [4], as well as `Sumatra` (sequence alignment) and `Sumaclust` (sequence alignment and clustering) [5] will be added, taking advantage of the database structure developed for the `OBITools3`.    
+
+
+
+**Implementation and disponibility.** The lower layers managing the DMS as well as all the compute-intensive functions are coded in `C99` for efficiency reasons. A `Cython` (<http://www.cython.org>) object layer allows for a simple but efficient implementation of the `OBITools3` commands in `Python 3.5`. The `OBITools3` are still in development, and the first functional versions are expected for autumn 2016.    
+
+
+
+**References.**
+
+1. Taberlet P, Coissac E, Hajibabaei M, Rieseberg LH: Environmental DNA. Mol Ecol 2012:1789–1793.
+2. Boyer F, Mercier C, Bonin A, Le Bras Y, Taberlet P, Coissac E: OBITools: a Unix-inspired software package for DNA metabarcoding. Mol Ecol Resour 2015:n/a–n/a.
+3. Riaz T, Shehzad W, Viari A, Pompanon F, Taberlet P, Coissac E: ecoPrimers: inference of new DNA barcode markers from whole genome sequence analysis. Nucleic Acids Res 2011, 39:e145.
+4. Ficetola GF, Coissac E, Zundel S, Riaz T, Shehzad W, Bessière J, Taberlet P, Pompanon F: An in silico approach for the evaluation of DNA barcodes. BMC Genomics 2010, 11:434.
+5. Mercier C, Boyer F, Bonin A, Coissac E (2013) SUMATRA and SUMACLUST: fast and exact comparison and clustering of sequences. Available: <http://metabarcoding.org/sumatra> and <http://metabarcoding.org/sumaclust>

distutils.ext/obidistutils/command/build_ext.py

@@ -7,107 +7,109 @@ Created on 13 fevr. 2014
 from distutils import log
 import os
 
-from Cython.Distutils import build_ext  as ori_build_ext  # @UnresolvedImport
-
-from Cython.Compiler import Options as cython_options  # @UnresolvedImport
 
 from distutils.errors import DistutilsSetupError
 
-class build_ext(ori_build_ext):
-
+try:
+    from Cython.Distutils import build_ext  as ori_build_ext  # @UnresolvedImport
+    from Cython.Compiler import Options as cython_options  # @UnresolvedImport
+    class build_ext(ori_build_ext):
     
-    def modifyDocScripts(self):
-        build_dir_file=open("doc/sphinx/build_dir.txt","w")
-        print(self.build_lib,file=build_dir_file)
-        build_dir_file.close()
         
-    def initialize_options(self):
-        ori_build_ext.initialize_options(self)  # @UndefinedVariable
-        self.littlebigman = None
-        self.built_files = None
-
-    
-    def finalize_options(self):
-        ori_build_ext.finalize_options(self)  # @UndefinedVariable
-
-        self.set_undefined_options('littlebigman',
-                                   ('littlebigman',  'littlebigman'))
-        
-        self.set_undefined_options('build_files',
-                                   ('files',  'built_files'))
-
-        self.cython_c_in_temp = 1
-                   
-        if self.littlebigman =='-DLITTLE_END':
-            if self.define is None:
-                self.define=[('LITTLE_END',None)]
-            else:
-                self.define.append('LITTLE_END',None)
-        
-    def substitute_sources(self,exe_name,sources):
-        """
-        Substitutes source file name starting by an @ by the actual
-        name of the built file (see --> build_files)
-        """
-        sources = list(sources)
-        for i in range(len(sources)):
-            message = "%s :-> %s" % (exe_name,sources[i])
-            if sources[i][0]=='@':
-                try:
-                    filename = self.built_files[sources[i][1:]]
-                except KeyError:
-                    tmpfilename = os.path.join(self.build_temp,sources[i][1:])
-                    if os.path.isfile       (tmpfilename):
-                        filename = tmpfilename
-                    else:
-                        raise DistutilsSetupError(
-                             'The %s filename declared in the source '
-                             'files of the program %s have not been '
-                             'built by the installation process' % (sources[i],
-                                                                     exe_name))
-                sources[i]=filename
-                log.info("%s changed to %s",message,filename)
-            else:
-                log.info("%s ok",message)
-
-        return sources
-
-    def build_extensions(self):
-        # First, sanity-check the 'extensions' list
-        
-        for ext in self.extensions:
-            ext.sources = self.substitute_sources(ext.name,ext.sources)
+        def modifyDocScripts(self):
+            build_dir_file=open("doc/sphinx/build_dir.txt","w")
+            print(self.build_lib,file=build_dir_file)
+            build_dir_file.close()
             
-        self.check_extensions_list(self.extensions)
-
-        for ext in self.extensions:
-            log.info("%s :-> %s",ext.name,ext.sources)
-            ext.sources = self.cython_sources(ext.sources, ext)
-            self.build_extension(ext)
-
-        
-    def run(self):
-        self.modifyDocScripts()
-
-        for cmd_name in self.get_sub_commands():
-            self.run_command(cmd_name)
-
-        cython_options.annotate = True
-        ori_build_ext.run(self)  # @UndefinedVariable
-        
-
-    def has_files(self):
-        return self.distribution.has_files()
-
-    def has_executables(self):
-        return self.distribution.has_executables()
+        def initialize_options(self):
+            ori_build_ext.initialize_options(self)  # @UndefinedVariable
+            self.littlebigman = None
+            self.built_files = None
     
-    sub_commands = [('build_files',has_files),
-                    ('build_cexe', has_executables)
-                    ] + \
-                   ori_build_ext.sub_commands
-
         
+        def finalize_options(self):
+            ori_build_ext.finalize_options(self)  # @UndefinedVariable
+    
+            self.set_undefined_options('littlebigman',
+                                       ('littlebigman',  'littlebigman'))
+            
+            self.set_undefined_options('build_files',
+                                       ('files',  'built_files'))
+    
+            self.cython_c_in_temp = 1
+                       
+            if self.littlebigman =='-DLITTLE_END':
+                if self.define is None:
+                    self.define=[('LITTLE_END',None)]
+                else:
+                    self.define.append('LITTLE_END',None)
+            
+        def substitute_sources(self,exe_name,sources):
+            """
+            Substitutes source file name starting by an @ by the actual
+            name of the built file (see --> build_files)
+            """
+            sources = list(sources)
+            for i in range(len(sources)):
+                message = "%s :-> %s" % (exe_name,sources[i])
+                if sources[i][0]=='@':
+                    try:
+                        filename = self.built_files[sources[i][1:]]
+                    except KeyError:
+                        tmpfilename = os.path.join(self.build_temp,sources[i][1:])
+                        if os.path.isfile       (tmpfilename):
+                            filename = tmpfilename
+                        else:
+                            raise DistutilsSetupError(
+                                 'The %s filename declared in the source '
+                                 'files of the program %s have not been '
+                                 'built by the installation process' % (sources[i],
+                                                                         exe_name))
+                    sources[i]=filename
+                    log.info("%s changed to %s",message,filename)
+                else:
+                    log.info("%s ok",message)
+    
+            return sources
+    
+        def build_extensions(self):
+            # First, sanity-check the 'extensions' list
+            
+            for ext in self.extensions:
+                ext.sources = self.substitute_sources(ext.name,ext.sources)
+                
+            self.check_extensions_list(self.extensions)
+    
+            for ext in self.extensions:
+                log.info("%s :-> %s",ext.name,ext.sources)
+                ext.sources = self.cython_sources(ext.sources, ext)
+                self.build_extension(ext)
+    
+            
+        def run(self):
+            self.modifyDocScripts()
+    
+            for cmd_name in self.get_sub_commands():
+                self.run_command(cmd_name)
+    
+            cython_options.annotate = True
+            ori_build_ext.run(self)  # @UndefinedVariable
+            
+    
+        def has_files(self):
+            return self.distribution.has_files()
+    
+        def has_executables(self):
+            return self.distribution.has_executables()
+        
+        sub_commands = [('build_files',has_files),
+                        ('build_cexe', has_executables)
+                        ] + \
+                       ori_build_ext.sub_commands
+
+except ImportError:
+    from distutils.command import build_ext  # @UnusedImport
+           
 
 
         

distutils.ext/obidistutils/serenity/bootstrappip.py

@@ -0,0 +1,36 @@
+'''
+Created on 22 janv. 2016
+
+@author: coissac
+'''
+
+import sys
+from urllib import request
+import os.path
+
+from obidistutils.serenity.util import get_serenity_dir
+from obidistutils.serenity.rerun import rerun_with_anothe_python
+from obidistutils.serenity.checkpython import is_a_virtualenv_python
+
+getpipurl="https://bootstrap.pypa.io/get-pip.py"
+
+def bootstrap():
+
+    getpipfile=os.path.join(get_serenity_dir(),"get-pip.py")
+    
+    with request.urlopen(getpipurl) as getpip:
+        with open(getpipfile,"wb") as out:
+            for l in getpip:
+                out.write(l)
+                
+    python = sys.executable
+    
+    if is_a_virtualenv_python():
+        command= "%s %s" % (python,getpipfile)        
+    else:
+        command= "%s %s --user" % (python,getpipfile)
+    
+    os.system(command)
+    
+    rerun_with_anothe_python(python)
+    

distutils.ext/obidistutils/serenity/checkpackage.py

@@ -7,8 +7,13 @@ Created on 2 oct. 2014
 import re
 import os
 
-import pip                                              # @UnresolvedImport
-from pip.utils import get_installed_distributions       # @UnresolvedImport
+try:
+    import pip                                              # @UnresolvedImport
+    from pip.utils import get_installed_distributions       # @UnresolvedImport
+except ImportError:
+    from .bootstrappip import bootstrap
+    bootstrap()
+
 from distutils.version import StrictVersion             # @UnusedImport
 from distutils.errors import DistutilsError
 from distutils import log

distutils.ext/obidistutils/serenity/virtual.py

@@ -59,7 +59,7 @@ def serenity_virtualenv(envname,package,version,minversion='3.4',maxversion=None
                     clear=True, 
                     symlinks=False, 
                     with_pip=True)
-        
+                
         # check the newly created virtualenv
         return serenity_virtualenv(envname,package,version)
     

python/obi.py

@@ -0,0 +1,50 @@
+#!/usr/local/bin/python3.4
+'''
+obi -- shortdesc
+
+obi is a description
+
+It defines classes_and_methods
+
+@author:     user_name
+
+@copyright:  2014 organization_name. All rights reserved.
+
+@license:    license
+
+@contact:    user_email
+@deffield    updated: Updated
+'''
+
+
+default_config = { 'software'       : "The OBITools",
+                   'log'            : False,
+                   'loglevel'       : 'INFO',
+                   'progress'       : True,
+                   'defaultdms'     : None
+                  }
+
+root_config_name='obi'
+
+from obitools3.apps.config import getConfiguration     # @UnresolvedImport
+from obitools3.version import version
+
+__all__     = []
+__version__ = version
+__date__    = '2014-09-28'
+__updated__ = '2014-09-28'
+
+DEBUG = 1
+TESTRUN = 0
+PROFILE = 0
+
+
+if __name__ =="__main__":
+    
+    config = getConfiguration(root_config_name,
+                              default_config)    
+                
+    config[root_config_name]['module'].run(config)
+    
+    
+    

python/obitools3/apps/arguments.pxd

@@ -0,0 +1,3 @@
+#cython: language_level=3
+
+cpdef buildArgumentParser(str configname, str softname)

python/obitools3/apps/arguments.pyx

@@ -0,0 +1,61 @@
+#cython: language_level=3
+
+'''
+Created on 27 mars 2016
+
+@author: coissac
+'''
+
+import argparse
+import sys
+
+
+from .command import getCommandsList
+
+class ObiParser(argparse.ArgumentParser): 
+    def error(self, message):
+        sys.stderr.write('error: %s\n' % message)
+        self.print_help()
+        sys.exit(2)
+
+cpdef buildArgumentParser(str configname, 
+                          str softname):
+    parser = ObiParser()
+    
+    parser.add_argument('--version',   dest='%s:version' % configname, 
+                                       action='store_true', 
+                                       default=False, 
+                        help='Print the version of %s' % softname)
+
+    parser.add_argument('--log',       dest='%s:log' % configname, 
+                                       action='store',
+                                       type=str,
+                                       default=None, 
+                        help='Create a logfile')
+
+    parser.add_argument('--no-progress', dest='%s:progress' % configname, 
+                                       action='store_false', 
+                                       default=None, 
+                        help='Do not print the progress bar during analyzes')
+
+    subparsers = parser.add_subparsers(title='subcommands',
+                                       description='valid subcommands',
+                                       help='additional help')
+    
+    commands = getCommandsList()
+    
+    for c in commands:
+        module = commands[c]
+        
+        if hasattr(module, "run"):
+            if hasattr(module, "__title__"):
+                sub = subparsers.add_parser(c,help=module.__title__)
+            else:
+                sub = subparsers.add_parser(c)
+    
+            if hasattr(module, "addOptions"):
+                module.addOptions(sub)
+            
+            sub.set_defaults(**{'%s:module'  % configname : module})
+                        
+    return parser

python/obitools3/apps/command.pxd

@@ -0,0 +1,3 @@
+#cython: language_level=3
+
+cdef object loadCommand(str name,loader)

python/obitools3/apps/command.pyx

@@ -0,0 +1,44 @@
+#cython: language_level=3
+
+'''
+Created on 27 mars 2016
+
+@author: coissac
+'''
+
+import pkgutil
+
+from obitools3 import commands
+
+cdef object loadCommand(str name,loader):
+    '''
+    Load a command module from its name and an ImpLoader
+    
+    This function is for internal use
+    
+    @param name:   name of the module
+    @type name: str 
+    @param loader: the module loader
+    @type loader: ImpLoader
+    
+    @return the loaded module
+    @rtype: module 
+    '''
+    
+    module = loader.find_module(name).load_module(name)
+    return module
+
+def getCommandsList():
+    '''
+    Returns the list of sub-commands available to the main `obi` command
+    
+    @return: a dict instance with key corresponding to each command and
+             value corresponding to the module
+             
+    @rtype: dict
+    '''
+    
+    cdef dict cmds = dict((x[1],loadCommand(x[1],x[0])) 
+                           for x in pkgutil.iter_modules(commands.__path__) 
+                           if not x[2])
+    return cmds

python/obitools3/apps/config.pxd

@@ -0,0 +1,10 @@
+#cython: language_level=3
+
+cpdef str setRootConfigName(str rootname)
+cpdef str getRootConfigName()
+
+cdef dict buildDefaultConfiguration(str root_config_name,
+                                    dict  config)
+
+cpdef dict getConfiguration(str root_config_name=?,
+                                    dict  config=?)

python/obitools3/apps/config.pyx

@@ -0,0 +1,103 @@
+#cython: language_level=3
+
+'''
+Created on 27 mars 2016
+
+@author: coissac
+'''
+
+import sys
+
+from .command   import  getCommandsList
+from .logging   cimport getLogger
+from .arguments cimport buildArgumentParser
+
+from ..version import version
+from _curses import version
+
+cdef dict __default_config__ = {}
+
+
+cpdef str setRootConfigName(str rootname):
+    global __default_config__
+    if '__root_config__' in __default_config__:
+        if __default_config__["__root_config__"] in __default_config__:
+            __default_config__[rootname]=__default_config__[__default_config__["__root_config__"]]
+            del __default_config__[__default_config__["__root_config__"]]
+    __default_config__['__root_config__']=rootname
+    return rootname
+
+cpdef str getRootConfigName():
+    global __default_config__
+    return __default_config__.get('__root_config__',None)
+
+cdef dict buildDefaultConfiguration(str root_config_name,
+                                    dict  config):
+    global __default_config__
+    
+    __default_config__.clear()
+    setRootConfigName(root_config_name)    
+
+    __default_config__[root_config_name]=config
+    
+    config['version']=version
+    
+    commands = getCommandsList()
+    
+    for c in commands:
+        module = commands[c]
+    
+        assert hasattr(module, "run")
+        
+        if hasattr(module, 'default_config'):
+            __default_config__[c]=module.default_config
+        else:
+            __default_config__[c]={}
+                        
+    return __default_config__
+        
+
+cpdef dict getConfiguration(str root_config_name="__default__",
+                            dict  config={}):
+    global __default_config__
+
+    if '__done__' in __default_config__:
+        return __default_config__
+    
+    if root_config_name=="__default__":
+        raise RuntimeError("No root_config_name specified")
+    
+    if not config:
+        raise RuntimeError("Base configuration is empty")
+    
+    
+    
+    config =  buildDefaultConfiguration(root_config_name,
+                                        config)
+    
+    parser = buildArgumentParser(root_config_name,
+                                 config[root_config_name]['software'])
+    
+    options = vars(parser.parse_args())
+
+    if options['%s:version' % root_config_name]:
+        print("%s - Version %s" % (config[root_config_name]['software'],
+                                   config[root_config_name]['version']))
+        sys.exit(0)
+    
+    for k in options:
+        section,key = k.split(':')
+        s = config[section]
+        if options[k] is not None:
+            s[key]=options[k]
+            
+    if not 'module' in config[root_config_name]:
+        print('\nError: No command specified',file=sys.stderr)
+        parser.print_help()
+        sys.exit(2)
+                
+    getLogger(config)
+    
+    config['__done__']=True
+            
+    return config

python/obitools3/apps/logging.pxd

@@ -0,0 +1,3 @@
+#cython: language_level=3
+
+cpdef getLogger(dict config)

python/obitools3/apps/logging.pyx

@@ -0,0 +1,46 @@
+#cython: language_level=3
+
+'''
+Created on 27 mars 2016
+
+@author: coissac
+'''
+
+import logging
+import sys
+
+cpdef getLogger(dict config):
+    '''
+    Returns the logger as defined by the command line option
+    or by the config file
+    :param config:
+    '''
+    
+    root = config["__root_config__"]
+ 
+    level  = config[root]['loglevel'] 
+    logfile= config[root]['log'] 
+    
+    rootlogger   = logging.getLogger()
+    logFormatter = logging.Formatter("%(asctime)s [%(levelname)-5.5s]  %(message)s")
+
+    stderrHandler = logging.StreamHandler(sys.stderr)
+    stderrHandler.setFormatter(logFormatter)
+
+    rootlogger.addHandler(stderrHandler)
+    
+    if logfile:
+        fileHandler = logging.FileHandler(logfile)
+        fileHandler.setFormatter(logFormatter)
+        rootlogger.addHandler(fileHandler)
+    
+    try:
+        loglevel = getattr(logging, level) 
+    except:
+        loglevel = logging.INFO
+        
+    rootlogger.setLevel(loglevel)
+    
+    config[root]['logger']=rootlogger
+    
+    return rootlogger

python/obitools3/apps/progress.pxd

@@ -0,0 +1,66 @@
+#cython: language_level=3
+
+from ..utils cimport str2bytes
+
+cdef extern from "stdio.h":
+    struct FILE
+    int fprintf(FILE *stream, char *format, ...)
+    int fputs(char *string, FILE *stream)
+    FILE* stderr
+    ctypedef unsigned int off_t "unsigned long long"
+    
+cdef extern from "unistd.h":
+    int fsync(int fd);
+
+cdef extern from "time.h":
+    struct tm :
+        int tm_yday 
+        int tm_hour
+        int tm_min
+        int tm_sec
+
+    enum: CLOCKS_PER_SEC
+    
+    ctypedef int time_t
+    ctypedef int clock_t
+    ctypedef int suseconds_t
+    
+    struct timeval:
+        time_t      tv_sec     #  seconds */
+        suseconds_t tv_usec    #  microseconds */
+
+
+    struct timezone :
+        int tz_minuteswest;    # minutes west of Greenwich
+        int tz_dsttime;        # type of DST correction 
+
+        
+    int gettimeofday(timeval *tv, timezone *tz)
+
+
+    tm *gmtime_r(time_t *clock, tm *result)
+    time_t time(time_t *tloc)
+    clock_t clock()
+
+cdef class ProgressBar:
+    cdef off_t   maxi
+    cdef clock_t starttime
+    cdef clock_t lasttime
+    cdef clock_t tickcount
+    cdef int freq
+    cdef int cycle
+    cdef int arrow
+    cdef int lastlog
+    cdef bint ontty
+    cdef int fd
+    
+    cdef bytes head
+    cdef char *chead
+    
+    cdef object logger
+
+    cdef char  *wheel
+    cdef char  *spaces
+    cdef char*  diese 
+    
+    cdef clock_t clock(self)

python/obitools3/apps/progress.pyx

@@ -0,0 +1,138 @@
+#cython: language_level=3
+
+'''
+Created on 27 mars 2016
+
+@author: coissac
+'''
+
+import sys
+from ..utils cimport bytes2str
+
+cdef class ProgressBar:
+    cdef clock_t clock(self):
+        cdef clock_t t
+        cdef timeval tp
+        cdef clock_t s
+        
+        <void> gettimeofday(&tp,NULL)
+        s = <clock_t> (<double> tp.tv_usec * 1.e-6 * <double> CLOCKS_PER_SEC)
+        t = tp.tv_sec * CLOCKS_PER_SEC + s 
+        
+        return t
+
+    def __init__(self,
+                 off_t maxi,
+                 dict  config,
+                 str head="",
+                 double seconde=0.1):
+        self.starttime = self.clock()
+        self.lasttime  = self.starttime
+        self.tickcount = <clock_t> (seconde * CLOCKS_PER_SEC)
+        self.freq      = 1
+        self.cycle     = 0
+        self.arrow     = 0
+        self.lastlog   = 0
+        
+        self.ontty = sys.stderr.isatty()
+        
+        if (maxi<=0):
+            maxi=1
+            
+        self.maxi = maxi
+        self.head = str2bytes(head)
+        self.chead= self.head 
+        
+        
+        self.logger=config[config["__root_config__"]]["logger"]
+        self.wheel =  '|/-\\'
+        self.spaces='          ' \
+                    '          ' \
+                    '          ' \
+                    '          ' \
+                    '          '
+        self.diese ='##########' \
+                    '##########' \
+                    '##########' \
+                    '##########' \
+                    '##########'  
+                           
+    def __call__(self,object pos):
+        cdef off_t    ipos
+        cdef clock_t  elapsed
+        cdef clock_t  newtime
+        cdef clock_t  delta
+        cdef clock_t  more 
+        cdef double   percent 
+        cdef tm remain
+        cdef int days,hour,minu,sec
+        cdef off_t fraction
+        cdef int twentyth
+        
+        self.cycle+=1
+    
+        if self.cycle % self.freq == 0:
+            self.cycle=1
+            newtime  = self.clock()
+            delta         = newtime - self.lasttime
+            self.lasttime = newtime
+            elapsed       = newtime - self.starttime
+#            print(" ",delta,elapsed,elapsed/CLOCKS_PER_SEC,self.tickcount)
+            
+            if   delta < self.tickcount / 5 :
+                self.freq*=2
+            elif delta > self.tickcount * 5 and self.freq>1:
+                self.freq/=2
+                
+            
+            if callable(pos):
+                ipos=pos()
+            else:
+                ipos=pos
+                
+            if ipos==0:
+                ipos=1                
+
+            percent = <double>ipos/<double>self.maxi
+            more = <time_t>((<double>elapsed / percent * (1. - percent))/CLOCKS_PER_SEC)
+            <void>gmtime_r(&more, &remain)
+            days  = remain.tm_yday 
+            hour  = remain.tm_hour
+            minu  = remain.tm_min
+            sec   = remain.tm_sec
+                
+            if self.ontty:
+                fraction=<int>(percent * 50.)
+                self.arrow=(self.arrow+1) % 4
+                self.diese[fraction]=0
+                self.spaces[50 - fraction]=0
+            
+                if days:
+                    <void>fprintf(stderr,b'\r%s %5.1f %% |%s%c%s] remain : %d days %02d:%02d:%02d',
+                                    self.chead,
+                                    percent*100,
+                                    self.diese,self.wheel[self.arrow],self.spaces,
+                                    days,hour,minu,sec)
+                else:
+                    <void>fprintf(stderr,b'\r%s %5.1f %% |%s%c%s] remain : %02d:%02d:%02d',
+                                    self.chead,
+                                    percent*100.,
+                                    self.diese,self.wheel[self.arrow],self.spaces,
+                                    hour,minu,sec)
+                self.diese[fraction]=b'#'
+                self.spaces[50 - fraction]=b' '
+
+            twentyth = int(percent * 20)
+            if twentyth != self.lastlog:
+                
+                if self.ontty:
+                    <void>fputs(b'\n',stderr)
+                    
+                self.logger.info('%s %5.1f %% remain : %02d:%02d:%02d' % (
+                                        bytes2str(self.head),
+                                        percent*100.,
+                                        hour,minu,sec))
+                self.lastlog=twentyth
+        else:
+            self.cycle+=1
+

python/obitools3/commands/count.py

@@ -0,0 +1,44 @@
+'''
+Created on 8 mars 2016
+
+@author: coissac
+'''
+
+from obitools3.apps.progress import ProgressBar  # @UnresolvedImport
+import time
+
+__title__="Counts sequences in a sequence set"
+
+
+default_config = {   'countmode'       : None
+                 }
+
+def addOptions(parser):
+    parser.add_argument(dest='obi:input',     metavar='obi:input', 
+                                              nargs='?', 
+                                              default=None,
+                        help='input data set' )
+
+    group=parser.add_argument_group('Obicount specific options')
+    group.add_argument('-s','--sequence',
+                             action="store_true", dest="count:sequence",
+                             default=False,
+                             help="Prints only the number of sequence records."
+                             )
+ 
+    group.add_argument('-a','--all',
+                             action="store_true", dest="count:all",
+                             default=False,
+                             help="Prints only the total count of sequence records (if a sequence has no `count` attribute, its default count is 1) (default: False)."
+                             )
+
+    
+    
+def run(config):
+    # The code of my command
+    pb = ProgressBar(1000,config,seconde=1)
+    
+    for i in range(1,1001):
+        pb(i)
+        time.sleep(0.01)
+        

python/obitools3/commands/import.pyx

@@ -0,0 +1,139 @@
+from obitools3.apps.progress cimport ProgressBar  # @UnresolvedImport
+from obitools3.files.universalopener cimport uopen
+from obitools3.parsers.fasta import fastaIterator
+from obitools3.parsers.fastq import fastqIterator
+from obitools3.obidms._obidms import OBIDMS
+
+
+import time
+
+__title__="Counts sequences in a sequence set"
+
+
+default_config = {   'destview'     : None,
+                     'skip'         : 0,
+                     'only'         : None,
+                     'skiperror'    : False,
+                     'seqinformat'  : None,
+                     'moltype'      : 'nuc',
+                     'filename'     : None
+                 }
+
+def addOptions(parser):
+    parser.add_argument(dest='import:filename',     
+                        metavar='<FILENAME>', 
+                        nargs='?', 
+                        default=None,
+                        help='sequence file name to be imported' )
+
+    group=parser.add_argument_group('obi import specific options')
+
+    group.add_argument('--default-dms','-d',
+                     action="store", dest="obi:defaultdms",
+                     metavar='<DMS NAME>',
+                     default=None,
+                     type=str,
+                     help="Name of the default DMS for reading and writing data")
+    
+    group.add_argument('--destination-view','-v',
+                     action="store", dest="import:destview",
+                     metavar='<VIEW NAME>',
+                     default=None,
+                     type=str,
+                     required=True,
+                     help="Name of the default DMS for reading and writing data")
+
+    group=parser.add_argument_group('obi import specific options')
+    
+    group.add_argument('--skip',
+                     action="store", dest="import:skip",
+                     metavar='<N>',
+                     default=None,
+                     type=int,
+                     help="skip the N first sequences")
+
+    group.add_argument('--only',
+                     action="store", dest="import:only",
+                     metavar='<N>',
+                     default=None,
+                     type=int,
+                     help="treat only N sequences")
+
+    group.add_argument('--skip-on-error',
+                     action="store_true", dest="import:skiperror",
+                     default=None,
+                     help="Skip sequence entries with parse error")
+    
+    group.add_argument('--fasta',
+                     action="store_const", dest="import:seqinformat",
+                     default=None,
+                     const='fasta',
+                     help="Input file is in fasta nucleic format (including obitools fasta extentions)")
+
+    group.add_argument('--fastq',
+                     action="store_const", dest="import:seqinformat",
+                     default=None,
+                     const='fastq',
+                     help="Input file is in sanger fastq nucleic format (standard fastq)")
+
+    group.add_argument('--nuc',
+                     action="store_const", dest="import:moltype",
+                     default=None,
+                     const='nuc',
+                     help="Input file contains nucleic sequences")
+    
+    group.add_argument('--prot',
+                     action="store_const", dest="import:moltype",
+                     default=None,
+                     const='pep',
+                     help="Input file contains protein sequences")
+
+    
+    
+def run(config):
+    pb = ProgressBar(35000000,config,seconde=5)
+      
+    inputs = uopen(config['import']['filename'])
+    
+    get_quality = False
+    if config['import']['seqinformat']=='fasta':
+        iseq = fastaIterator(inputs)
+        view_type="NUC_SEQS_VIEW"
+    elif config['import']['seqinformat']=='fastq':
+        iseq = fastqIterator(inputs)
+        view_type="NUC_SEQS_VIEW"
+        get_quality = True
+    else:
+        raise RuntimeError('No file format specified')
+    
+    # Temporary way to handle NA values
+    #NA_list = ["nan"]
+    
+    # Create DMS
+    d = OBIDMS(config['obi']['defaultdms'])
+    
+    # Create view
+    view = d.new_view(config['import']['destview'], view_type=view_type)
+    
+    i = 0
+    for seq in iseq:
+        pb(i)
+        view[i].set_id(seq['id'])
+        view[i].set_definition(seq['definition'])
+        view[i].set_sequence(seq['sequence'])
+        if get_quality :
+            view[i].set_quality(seq['quality'])
+        for tag in seq['tags'] :
+            #print(tag, seq['tags'][tag])
+            #if seq['tags'][tag] not in NA_list :
+            view[i][tag] = seq['tags'][tag]
+        i+=1
+
+    #print(i)
+    print(view.__repr__())
+
+    view.save_and_close()
+    d.close()
+    
+    print("Done.")
+        

python/obitools3/files/linebuffer.pxd

@@ -0,0 +1,6 @@
+#cython: language_level=3
+
+
+cdef class LineBuffer:
+    cdef object fileobj
+    cdef int    size

python/obitools3/files/linebuffer.pyx

@@ -0,0 +1,23 @@
+#cython: language_level=3
+
+'''
+Created on 30 mars 2016
+
+@author: coissac
+'''
+
+cdef class LineBuffer:
+
+    def __init__(self,object fileobj,int size=100000000):
+        self.fileobj=fileobj 
+        self.size=size
+    
+    def __iter__(self):
+        cdef list buff = self.fileobj.readlines(self.size)
+        cdef str  l
+        
+        while buff:
+            for l in buff:
+                yield l
+            buff = self.fileobj.readlines(self.size)
+    

python/obitools3/files/uncompress.pxd

@@ -0,0 +1,18 @@
+#cython: language_level=3
+
+cdef class MagicKeyFile:
+    cdef object stream
+    cdef str    stream_mode
+    cdef object binary
+    cdef bytes  key 
+    cdef int    keylength 
+    cdef int    pos
+    
+    cpdef bytes read(self,int size=?)
+    cpdef bytes read1(self,int size=?)
+    cpdef int tell(self)
+
+    
+cdef class CompressedFile:
+    cdef object accessor
+    

python/obitools3/files/uncompress.pyx

@@ -0,0 +1,117 @@
+#cython: language_level=3
+
+'''
+Created on 28 mars 2016
+
+@author: coissac
+'''
+
+import zipfile
+import bz2
+import gzip
+
+import io
+
+cdef class MagicKeyFile:
+    def __init__(self,stream,length=2):
+        
+        binary=stream
+        self.stream = stream
+        self.stream_mode = None 
+        if hasattr(stream, "mode"):
+            self.stream_mode = stream.mode
+            if (not 'b' in stream.mode and 
+                hasattr(stream, "buffer") and 
+                'b' in stream.buffer.mode):
+                binary=stream.buffer
+
+        if (self.stream_mode is None and 
+            not (hasattr(stream, 'headers') and 
+                 hasattr(stream.headers, "keys") and
+                 'Content-type' in stream.headers)):
+            raise TypeError("stream does not present the good interface")
+        
+        self.binary=binary
+        self.key=binary.read(length)
+        self.keylength=length
+        self.pos=0
+        
+    cpdef bytes read(self,int size=-1):
+        cdef bytes r
+        
+        if self.pos < self.keylength:
+            if size > (self.keylength - self.pos):
+                size = size - self.keylength + self.pos
+                r = self.key[self.pos:] + self.binary.read(size)
+                self.pos=self.keylength + 1
+            elif size >=0 :
+                r = self.key[self.pos:(self.pos+size)]
+                self.pos+=size
+            else:
+                r = self.key[self.pos:] + self.binary.read(size)
+                self.pos=self.keylength + 1
+        else:
+            r = self.binary.read(size)
+            
+        return r
+
+    cpdef bytes read1(self,int size=-1):
+        return self.read(size)
+
+    cpdef int tell(self):
+        cdef int p
+        
+        if self.pos < self.keylength:
+            p = self.pos 
+        else:
+            p = self.binary.tell()
+        
+        return p
+    
+    def __getattr__(self,name):
+        return getattr(self.binary, name)
+        
+
+    
+cdef class CompressedFile:
+    
+    
+    def __init__(self,stream):
+        cdef int            keylength
+        cdef MagicKeyFile   magic
+        cdef str            compressor
+        cdef bytes          k 
+        cdef object         c
+        
+        cdef dict compress = { 'zip' : (b'\x50\x4b\x03\x04',zipfile.ZipFile),
+                               'bz2' : (b'\x42\x5a\x68',bz2.BZ2File),
+                               'gz'  : (b'\x1f\x8b\x08',gzip.open)
+                   }
+        
+        keylength = max([len(x[0]) for x in compress.values()])
+        magic=MagicKeyFile(stream,keylength)
+        
+        self.accessor = None
+        
+        for compressor in compress:
+            k,c = compress[compressor]
+            if magic.key.startswith(k):
+                self.accessor = c(magic)
+        
+        if self.accessor is None:
+            self.accessor = magic
+        
+        if ((hasattr(stream, 'headers') and 
+             hasattr(stream.headers, "keys") and
+             'Content-type' in stream.headers and
+             stream.headers['Content-type'].startswith('text/')) or
+            'b' not in magic.stream_mode):
+            self.accessor = io.TextIOWrapper(self.accessor)
+        
+    
+    def __getattr__(self,name):
+        return getattr(self.accessor, name)
+    
+    def __iter__(self):
+        for x in self.accessor:
+            yield x 

python/obitools3/files/universalopener.pxd

@@ -0,0 +1,5 @@
+#cython: language_level=3
+
+from .uncompress cimport CompressedFile
+
+cpdef CompressedFile uopen(str name, mode=?)

python/obitools3/files/universalopener.pyx

@@ -0,0 +1,23 @@
+#cython: language_level=3
+
+'''
+Created on 25 mars 2016
+
+@author: coissac
+'''
+
+from urllib.request import urlopen
+
+
+cpdef CompressedFile uopen(str name, mode='r'):
+    cdef CompressedFile c
+    
+    try:
+        f = urlopen(name)
+    except:
+        f = open(name,mode) 
+        
+    c = CompressedFile(f)
+    
+    return c
+    

python/obitools3/format/fasta.pxd

@@ -0,0 +1,10 @@
+from ..utils cimport bytes2str
+from .header cimport HeaderFormat
+from cython.view cimport array as cvarray
+
+cdef class FastaFormat:
+
+    cdef HeaderFormat headerFormater
+    
+    cdef size_t  sequenceBufferLength
+    cdef char*   sequenceBuffer

python/obitools3/format/fasta.pyx

@@ -0,0 +1,32 @@
+cimport cython
+from libc.stdlib cimport malloc, free, realloc
+from libc.string cimport strncpy
+
+cdef class FastaFormat:
+	
+	def __init__(self, list tags=[], bint printNAKeys=False):
+		self.headerFormater = HeaderFormat(True,
+										   tags,
+										   printNAKeys)
+		
+	@cython.boundscheck(False)	
+	def __call__(self, dict data):
+		cdef bytes brawseq      = data['sequence']
+		cdef size_t lseq        = len(brawseq)
+		cdef size_t k=0
+		cdef list   lines       = []
+		
+		for k in range(0,lseq,60):
+			lines.append(brawseq[k:(k+60)])
+			
+		brawseq = b'\n'.join(lines)
+		
+		return "%s\n%s" % (self.headerFormater(data),bytes2str(brawseq))
+					
+		
+				
+			
+		
+		
+		
+		

python/obitools3/format/header.pxd

@@ -0,0 +1,7 @@
+cdef class HeaderFormat:
+
+    cdef str    start
+    cdef set    tags
+    cdef bint   printNaKeys
+    cdef size_t headerBufferLength
+    

python/obitools3/format/header.pyx

@@ -0,0 +1,60 @@
+
+cdef class HeaderFormat:
+    
+    def __init__(self, bint fastaHeader=True, list tags=[], bint printNAKeys=False):
+        '''
+        
+            @param fastaHeader: 
+            @type  fastaHeader: `bool`
+            
+            @param tags:
+            @type  tags: `list` of `bytes`
+            
+            @param printNAKeys: 
+            @type  printNAKeys: `bool` 
+        '''
+        
+        self.tags           = set(tags)
+        self.printNaKeys    = printNAKeys
+        
+        if fastaHeader:
+            self.start=">"
+        else:
+            self.start="@"
+        
+        self.headerBufferLength = 1000
+        #self.headerBuffer = []
+        
+    def __call__(self, dict data):
+        cdef str header
+        cdef dict tags = data['tags']
+        cdef set  ktags
+        cdef list lines = [""]
+        cdef str tagline
+        
+        if self.tags is not None and self.tags:
+            ktags = self.tags
+        else:
+            ktags = set(tags.keys())
+            
+        for k in ktags:
+            if k in tags:
+                value = tags[k]
+                if value is not None or self.printNaKeys:
+                    lines.append("%s=%s;" % (k,tags[k]))
+                    
+        if len(lines) > 1:
+            tagline=" ".join(lines)
+        else:
+            tagline=""
+            
+        if data['definition'] is not None:
+            header = "%s%s%s %s" % (self.start,data['id'],
+                                    tagline,
+                                    data['definition'])
+        else:
+            header = "%s%s%s" % (self.start,data['id'],
+                                 tagline)
+            
+        return header
+        

python/obitools3/obidms/_obidms.cfiles

@@ -1,5 +1,44 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
 ../../../src/obidms.h
 ../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
@@ -10,7 +49,11 @@
 ../../../src/obilittlebigman.c
 ../../../src/obitypes.h
 ../../../src/obitypes.c
-../../../src/private_at_functions.h
-../../../src/private_at_functions.c
-../../../src/obiarray.h
-../../../src/obiarray.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
+../../../src/utils.h
+../../../src/utils.c

python/obitools3/obidms/_obidms.pxd

@@ -2,46 +2,94 @@
 
 from .capi.obidms       cimport OBIDMS_p
 from .capi.obidmscolumn cimport OBIDMS_column_p
+from .capi.obiview      cimport Obiview_p
 from .capi.obitypes     cimport obiversion_t, OBIType_t, index_t
+from ._obitaxo          cimport OBI_Taxonomy
 
 
-cdef class OBIDMS_column
+cdef class OBIDMS_column:
+
+    cdef  OBIDMS_column_p*   pointer
+    cdef  OBIDMS             dms
+    cdef  OBIView            view
+    cdef  str                data_type
+    cdef  str                dms_name
+    cdef  str                column_name
+    cdef  index_t            nb_elements_per_line
+    cdef  list               elements_names
+        
+    cpdef         update_pointer(self)
+    cpdef list    get_elements_names(self)
+    cpdef str     get_data_type(self)
+    cpdef index_t get_nb_lines_used(self)
+    cpdef str     get_creation_date(self)
+    cpdef str     get_comments(self)
+    cpdef         close(self)
+    @staticmethod
+    cdef  object  get_subclass_type(OBIDMS_column_p column_p)
+
+
+cdef class OBIDMS_column_multi_elts(OBIDMS_column):
+    
+    cpdef set_line(self, index_t line_nb, dict values)
+
+
+cdef class OBIDMS_column_line:
+
+    cdef OBIDMS_column column
+    cdef index_t index
+
+
+cdef class OBIView:
+    
+    cdef Obiview_p pointer
+    cdef str       name
+    cdef str       comments
+    cdef dict      columns
+    cdef OBIDMS    dms
+    
+    cpdef delete_column(self, str column_name)
+    cpdef add_column(self, 
+                     str column_name, 
+                     obiversion_t version_number=*, 
+                     str type=*,
+                     index_t nb_lines=*, 
+                     index_t nb_elements_per_line=*,
+                     list elements_names=*,
+                     str indexer_name=*,
+                     str comments=*,
+                     bint create=*
+                    )
+    cpdef select_line(self, index_t line_nb)
+    cpdef select_lines(self, list line_selection)
+    cpdef save_and_close(self)
+
+
+cdef class OBIView_NUC_SEQS(OBIView):
+
+    cdef OBIDMS_column ids
+    cdef OBIDMS_column sequences
+    cdef OBIDMS_column definitions
+    cdef OBIDMS_column qualities
+    
+    cpdef delete_column(self, str column_name)
+    
+
+cdef class OBIView_line :
+
+    cdef index_t index
+    cdef OBIView view
+
 
-            
 cdef class OBIDMS:
 
     cdef OBIDMS_p pointer
     cdef str dms_name
     
-    cpdef dict list(self)
     cpdef close(self)
-    cpdef OBIDMS_column open_column(self,
-                                    str column_name,
-                                    bint create=*, 
-                                    bint clone=*, bint clone_data=*,
-                                    obiversion_t version_number=*, 
-                                    OBIType_t data_type=*,
-                                    index_t nb_lines=*, 
-                                    index_t nb_elements_per_line=*, 
-                                    list elements_names=*,
-                                    str array_name=*)
-
-
-cdef class OBIDMS_column:
-
-    cdef OBIDMS_column_p    pointer
-    cdef OBIDMS             dms
-    cdef str                data_type   # TODO keep as OBIType_t? both?
-    cdef str                dms_name
-    cdef str                column_name
-    cdef index_t             nb_elements_per_line
-    cdef list               elements_names
-
-#    cpdef object get_item(self, index_t line_nb, str element_name)                  TODO can't declare because not the same in all subclasses
-#    cpdef        set_item(self, index_t line_nb, str element_name, object value)    TODO can't declare because object value
-    cpdef list    get_elements_names(self)
-    cpdef str     get_data_type(self)
-    cpdef index_t get_nb_lines_used(self)
-    cpdef str     get_creation_date(self)
-    cpdef         close(self)
+    cpdef OBI_Taxonomy open_taxonomy(self, str taxo_name)
+    cpdef OBIView open_view(self, str view_name)
+    cpdef OBIView new_view(self, str view_name, object view_to_clone=*, list line_selection=*, str view_type=*, str comments=*)
+    cpdef dict read_view_infos(self, str view_name)
+#    cpdef dict read_views(self)    TODO
 

python/obitools3/obidms/_obidms.pyx

@@ -1,52 +1,569 @@
 #cython: language_level=3
 
-from pathlib import Path
-
 from obitools3.utils cimport bytes2str, str2bytes
 
 from .capi.obidms cimport obi_dms, \
                           obi_close_dms
-from .capi.obidmscolumn cimport obi_column_get_header_from_name, \
-                                obi_unmap_header, \
-                                obi_column_get_latest_version_from_name, \
-                                obi_create_column, \
-                                obi_clone_column, \
-                                obi_open_column, \
-                                obi_close_column, \
-                                obi_column_format_date, \
+                          
+from .capi.obidmscolumn cimport obi_close_column, \
+                                OBIDMS_column_p, \
                                 OBIDMS_column_header_p
-from .capi.obitypes cimport const_char_p, \
-                            name_data_type
-                            
 
-from ._obidms cimport OBIDMS
-from ._obidms cimport OBIDMS_column
+from .capi.obiutils cimport obi_format_date
+                                
+from .capi.obitypes cimport const_char_p, \
+                            OBIType_t, \
+                            OBI_INT, \
+                            OBI_FLOAT, \
+                            OBI_BOOL, \
+                            OBI_CHAR, \
+                            OBI_QUAL, \
+                            OBI_STR, \
+                            OBI_SEQ, \
+                            name_data_type, \
+                            only_ATGC    # discuss
+                            
+from ._obidms cimport OBIDMS, \
+                      OBIDMS_column, \
+                      OBIView, \
+                      OBIView_line
+
+from ._obitaxo cimport OBI_Taxonomy
+
+from ._obiseq cimport OBI_Nuc_Seq, OBI_Nuc_Seq_Stored
 
 from ._obidmscolumn_int cimport OBIDMS_column_int, \
-                                OBIDMS_column_int_writable, \
-                                OBIDMS_column_int_multi_elts, \
-                                OBIDMS_column_int_multi_elts_writable
+                                OBIDMS_column_multi_elts_int
                                 
 from ._obidmscolumn_float cimport OBIDMS_column_float, \
-                                  OBIDMS_column_float_writable, \
-                                  OBIDMS_column_float_multi_elts, \
-                                  OBIDMS_column_float_multi_elts_writable
+                                  OBIDMS_column_multi_elts_float
 
 from ._obidmscolumn_bool cimport OBIDMS_column_bool, \
-                                 OBIDMS_column_bool_writable, \
-                                 OBIDMS_column_bool_multi_elts, \
-                                 OBIDMS_column_bool_multi_elts_writable
+                                 OBIDMS_column_multi_elts_bool
 
 from ._obidmscolumn_char cimport OBIDMS_column_char, \
-                                 OBIDMS_column_char_writable, \
-                                 OBIDMS_column_char_multi_elts, \
-                                 OBIDMS_column_char_multi_elts_writable
+                                 OBIDMS_column_multi_elts_char
+
+from ._obidmscolumn_qual cimport OBIDMS_column_qual, \
+                                 OBIDMS_column_multi_elts_qual
 
 from ._obidmscolumn_str cimport OBIDMS_column_str, \
-                                OBIDMS_column_str_writable, \
-                                OBIDMS_column_str_multi_elts, \
-                                OBIDMS_column_str_multi_elts_writable
+                                OBIDMS_column_multi_elts_str
 
+from ._obidmscolumn_seq cimport OBIDMS_column_seq, \
+                                OBIDMS_column_multi_elts_seq
+
+from .capi.obiview cimport Obiview_p, \
+                           Obiview_infos_p, \
+                           Column_reference_p, \
+                           obi_new_view_nuc_seqs, \
+                           obi_new_view, \
+                           obi_new_view_cloned_from_name, \
+                           obi_new_view_nuc_seqs_cloned_from_name, \
+                           obi_view_map_file, \
+                           obi_view_unmap_file, \
+                           obi_open_view,  \
+                           obi_view_delete_column, \
+                           obi_view_add_column, \
+                           obi_view_get_column, \
+                           obi_view_get_column, \
+                           obi_view_get_pointer_on_column_in_view, \
+                           obi_select_line, \
+                           obi_select_lines, \
+                           obi_save_and_close_view, \
+                           VIEW_TYPE_NUC_SEQS, \
+                           NUC_SEQUENCE_COLUMN, \
+                           ID_COLUMN, \
+                           DEFINITION_COLUMN, \
+                           QUALITY_COLUMN
+                           
+from libc.stdlib cimport malloc
+from cpython.pycapsule cimport PyCapsule_New, PyCapsule_GetPointer
+
+
+cdef class OBIDMS_column :
+
+    # Should only be initialized through a subclass
+    def __init__(self, OBIView view, str column_name):
+        
+        cdef OBIDMS_column_p  column_p
+        cdef OBIDMS_column_p* column_pp
+        
+        column_pp = obi_view_get_pointer_on_column_in_view(view.pointer, str2bytes(column_name))
+        column_p = column_pp[0]     # TODO ugly cython dereferencing but can't find better
+        
+        # Fill structure
+        self.pointer = column_pp
+        self.dms = view.dms
+        self.view = view
+        self.data_type = bytes2str(name_data_type((column_p.header).returned_data_type))
+        self.column_name = bytes2str((column_p.header).name)
+        self.nb_elements_per_line = (column_p.header).nb_elements_per_line
+        self.elements_names = (bytes2str((column_p.header).elements_names)).split(';')
+
+    def __setitem__(self, index_t line_nb, object value):
+        self.set_line(line_nb, value)
+
+    def __getitem__(self, index_t line_nb):
+        return self.get_line(line_nb)
+
+    def __len__(self):
+        return (self.pointer)[0].header.lines_used
+    
+    def __sizeof__(self):
+        return ((self.pointer)[0].header.header_size + (self.pointer)[0].header.data_size)
+    
+    def __iter__(self):
+    
+        # Declarations
+        cdef index_t lines_used
+        cdef index_t line_nb
+        
+        # Yield each line
+        lines_used = (self.pointer)[0].header.lines_used
+        for line_nb in range(lines_used):
+            yield self.get_line(line_nb)
+    
+    cpdef update_pointer(self):
+        self.pointer = <OBIDMS_column_p*> obi_view_get_pointer_on_column_in_view(self.view.pointer, str2bytes(self.column_name))
+
+    cpdef list get_elements_names(self):
+        return self.elements_names
+         
+    cpdef str get_data_type(self):
+        return self.data_type
+    
+    cpdef index_t get_nb_lines_used(self):
+        return (self.pointer)[0].header.lines_used
+        
+    cpdef str get_creation_date(self):
+        return bytes2str(obi_format_date((self.pointer)[0].header.creation_date))
+    
+    cpdef str get_comments(self):
+        return bytes2str((self.pointer)[0].header.comments)
+    
+    def __str__(self) :
+        cdef str to_print
+        to_print = ''
+        for line in self :
+            to_print = to_print + str(line) + "\n"
+        return to_print
+
+    def __repr__(self) :
+        return (self.column_name + ", version " + str((self.pointer)[0].header.version) + ", data type: " + self.data_type)
+        
+    cpdef close(self):
+        if obi_close_column((self.pointer)[0]) < 0 :
+            raise Exception("Problem closing a column")
+
+
+    @staticmethod
+    cdef object get_subclass_type(OBIDMS_column_p column_p) :
+        
+        cdef object subclass
+        cdef OBIDMS_column_header_p header
+        cdef OBIType_t col_type
+        cdef bint col_writable
+        cdef bint col_one_element_per_line
+
+        header = column_p.header        
+        col_type = header.returned_data_type
+        col_writable = column_p.writable
+        col_one_element_per_line = ((header.nb_elements_per_line) == 1)
+        
+        if col_type == OBI_INT :
+            if col_one_element_per_line :
+                subclass = OBIDMS_column_int
+            else :
+                subclass = OBIDMS_column_multi_elts_int
+        elif col_type == OBI_FLOAT :
+            if col_one_element_per_line :
+                subclass = OBIDMS_column_float
+            else :
+                subclass = OBIDMS_column_multi_elts_float
+        elif col_type == OBI_BOOL :
+            if col_one_element_per_line :
+                subclass = OBIDMS_column_bool
+            else :
+                subclass = OBIDMS_column_multi_elts_bool
+        elif col_type == OBI_CHAR :
+            if col_one_element_per_line :
+                subclass = OBIDMS_column_char
+            else :
+                subclass = OBIDMS_column_multi_elts_char
+        elif col_type == OBI_QUAL :
+            if col_one_element_per_line :
+                subclass = OBIDMS_column_qual
+            else :
+                subclass = OBIDMS_column_multi_elts_qual
+        elif col_type == OBI_STR :
+            if col_one_element_per_line :
+                subclass = OBIDMS_column_str
+            else :
+                subclass = OBIDMS_column_multi_elts_str
+        elif col_type == OBI_SEQ :
+            if col_one_element_per_line :
+                subclass = OBIDMS_column_seq
+            else :
+                subclass = OBIDMS_column_multi_elts_seq
+        else :
+            raise Exception("Problem with the data type")
+
+        return subclass
+
+    
+######################################################################################################
+
+cdef class OBIDMS_column_multi_elts(OBIDMS_column) :
+    
+    def __getitem__(self, index_t line_nb):
+        return OBIDMS_column_line(self, line_nb)
+
+    cpdef set_line(self, index_t line_nb, dict values):
+        for element_name in values :
+            self.set_item(line_nb, element_name, values[element_name])
+
+######################################################################################################
+
+cdef class OBIDMS_column_line :
+
+    def __init__(self, OBIDMS_column column, index_t line_nb) :
+        self.index = line_nb
+        self.column = column
+        
+    def __getitem__(self, str element_name) :
+        return self.column.get_item(self.index, element_name)
+     
+    def __setitem__(self, str element_name, object value):
+        self.column.set_item(self.index, element_name, value)
+
+    def __contains__(self, str element_name):
+        return (element_name in self.column.elements_names)
+
+    def __repr__(self) :
+        return str(self.column.get_line(self.index))
+
+##########################################
+
+
+cdef class OBIView :
+    
+    def __init__(self, OBIDMS dms, str view_name, bint new=False, object view_to_clone=None, list line_selection=None, str comments=""): 
+                
+        cdef Obiview_p              view = NULL
+        cdef int                    i
+        cdef list                   col_list
+        cdef str                    col_name
+        cdef OBIDMS_column          column
+        cdef OBIDMS_column_p        column_p
+        cdef OBIDMS_column_header_p header
+        cdef index_t*               line_selection_p
+        
+        self.dms = dms
+        
+        # Create the C array for the line selection if needed
+        if line_selection is not None :
+            line_selection_p = <index_t*> malloc((len(line_selection) + 1) * sizeof(index_t))
+            for i in range(len(line_selection)) :
+                line_selection_p[i] = line_selection[i]
+            line_selection_p[len(line_selection)] = -1
+        else :
+            line_selection_p = NULL
+       
+        # Create the view if needed
+        if new :
+            if view_to_clone is not None :
+                if type(view_to_clone) == str :
+                    view = obi_new_view_cloned_from_name(dms.pointer, str2bytes(view_name), str2bytes(view_to_clone), line_selection_p, str2bytes(comments))
+                else :
+                    view = obi_new_view(dms.pointer, str2bytes(view_name), (<OBIView> view_to_clone).pointer, line_selection_p, str2bytes(comments))
+            elif view_to_clone is None :
+                view = obi_new_view(dms.pointer, str2bytes(view_name), NULL, line_selection_p, str2bytes(comments))
+        # Else, open the existing view
+        elif not new :
+            if view_name is not None :
+                view = obi_open_view(dms.pointer, str2bytes(view_name))
+            elif view_name is None :
+                view = obi_open_view(dms.pointer, NULL) # TODO discuss
+        
+        if view == NULL :
+            raise Exception("Error creating/opening a view")
+
+        self.pointer = view
+        self.name = bytes2str(view.infos.name)
+        
+        # Go through columns to build list of corresponding python instances
+        self.columns = {}
+        for i in range(view.infos.column_count) :
+            column_p               = <OBIDMS_column_p> (view.columns)[i]
+            header                 = (column_p).header
+            col_name               = bytes2str(header.name)         
+            subclass               = OBIDMS_column.get_subclass_type(column_p)
+            self.columns[col_name] = subclass(self, col_name)
+
+
+    def __repr__(self) :
+        cdef str s
+        s = str(self.name) + ", " + str(self.comments) + ", " + str(self.pointer.infos.line_count) + " lines\n" 
+        for column_name in self.columns :
+            s = s + self.columns[column_name].__repr__() + '\n'
+        return s
+        
+    
+    cpdef delete_column(self, str column_name) :
+        
+        cdef int i
+        cdef Obiview_p view_p
+        cdef OBIDMS_column column
+        cdef OBIDMS_column_p column_p
+        cdef OBIDMS_column_header_p header
+        cdef str column_n
+ 
+        view = self.pointer
+ 
+        if obi_view_delete_column(view_p, str2bytes(column_name)) < 0 :
+            raise Exception("Problem deleting a column from a view")
+        
+        # Update the dictionaries of column pointers and column objects, and update pointers in column objects (make function?):
+        (self.columns).pop(column_name)
+        
+        for column_n in self.columns :
+            (self.columns[column_n]).update_pointer()
+    
+    
+    cpdef add_column(self, 
+                     str column_name, 
+                     obiversion_t version_number=-1, 
+                     str type='',
+                     index_t nb_lines=0, 
+                     index_t nb_elements_per_line=1,
+                     list elements_names=None,
+                     str indexer_name="",
+                     str comments="",
+                     bint create=True
+                     ) :
+        
+        cdef bytes column_name_b
+        cdef bytes elements_names_b
+        cdef object subclass
+        cdef OBIDMS_column_p column_p
+        
+        column_name_b = str2bytes(column_name)
+        
+        if nb_elements_per_line > 1 :
+            elements_names_b = str2bytes(';'.join(elements_names))
+        elif nb_elements_per_line == 1 :
+            elements_names_b = column_name_b
+    
+        if type :    
+            if type == 'OBI_INT' :
+                data_type = OBI_INT
+            elif type == 'OBI_FLOAT' :
+                data_type = OBI_FLOAT
+            elif type == 'OBI_BOOL' :
+                data_type = OBI_BOOL
+            elif type == 'OBI_CHAR' :
+                data_type = OBI_CHAR
+            elif type == 'OBI_QUAL' :
+                data_type = OBI_QUAL
+            elif type == 'OBI_STR' :
+                data_type = OBI_STR
+            elif type == 'OBI_SEQ' :
+                data_type = OBI_SEQ
+            else :
+                raise Exception("Invalid provided data type")
+        
+        if (obi_view_add_column(self.pointer, column_name_b, version_number,     # TODO should return pointer on column?
+                                data_type, nb_lines, nb_elements_per_line, 
+                                elements_names_b, str2bytes(indexer_name), 
+                                str2bytes(comments), create) < 0) :
+            raise Exception("Problem adding a column in a view")
+        
+        # Get the column pointer
+        column_p = obi_view_get_column(self.pointer, column_name_b)
+        
+        # Open and store the subclass
+        subclass = OBIDMS_column.get_subclass_type(column_p)
+        (self.columns)[column_name] = subclass(self, column_name)
+                
+    
+    cpdef save_and_close(self) :
+        if (obi_save_and_close_view(self.pointer) < 0) :
+            raise Exception("Problem closing a view")
+    
+
+    def __iter__(self):
+        # iter on each line of all columns
+        
+        # Declarations
+        cdef index_t      lines_used
+        cdef index_t      line_nb
+        cdef OBIView_line line          # TODO Check that this works for NUC SEQ views
+        
+        # Yield each line
+        lines_used = self.pointer.infos.line_count
+        
+        for line_nb in range(lines_used) :
+            line = self[line_nb]
+            yield line
+
+
+    def __getitem__(self, object item) :
+        if type(item) == str :
+            return (self.columns)[item]
+        elif type(item) == int :
+            return OBIView_line(self, item)
+
+
+    cpdef select_line(self, index_t line_nb) :
+        if obi_select_line(self.pointer, line_nb) < 0 :
+            raise Exception("Problem selecting a line")
+
+    
+    cpdef select_lines(self, list line_selection) :
+        cdef index_t* line_selection_p
+        line_selection_p = <index_t*> malloc((len(line_selection) + 1) * sizeof(index_t))
+        for i in range(len(line_selection)) :
+            line_selection_p[i] = line_selection[i]
+        line_selection_p[len(line_selection)] = -1
+        if obi_select_lines(self.pointer, line_selection_p) < 0 :
+            raise Exception("Problem selecting a list of lines")
+    
+    
+    def __contains__(self, str column_name):
+        return (column_name in self.columns)
+    
+    
+    def __str__(self) :
+        cdef OBIView_line line
+        cdef str to_print
+        to_print = ""
+        for line in self.__iter__() :
+            to_print = to_print + str(line) + "\n"
+        return to_print
+
+#############################################
+
+cdef class OBIView_NUC_SEQS(OBIView):
+    
+    def __init__(self, OBIDMS dms, str view_name, bint new=False, object view_to_clone=None, list line_selection=None, str comments=""): 
+        
+        cdef Obiview_p              view = NULL
+        cdef int                    i
+        cdef list                   col_list
+        cdef str                    col_name
+        cdef OBIDMS_column          column
+        cdef OBIDMS_column_p        column_p
+        cdef OBIDMS_column_header_p header
+        cdef index_t*               line_selection_p
+                
+        self.dms = dms
+        
+        if line_selection is not None :
+            line_selection_p = <index_t*> malloc((len(line_selection) + 1) * sizeof(index_t))
+            for i in range(len(line_selection)) :
+                line_selection_p[i] = line_selection[i]
+            line_selection_p[len(line_selection)] = -1
+        else :
+            line_selection_p = NULL
+       
+        if new :
+            if view_to_clone is not None :
+                if type(view_to_clone) == str :
+                    view = obi_new_view_nuc_seqs_cloned_from_name(dms.pointer, str2bytes(view_name), str2bytes(view_to_clone), line_selection_p, str2bytes(comments))
+                else :
+                    view = obi_new_view_nuc_seqs(dms.pointer, str2bytes(view_name), (<OBIView> view_to_clone).pointer, line_selection_p, str2bytes(comments))
+            elif view_to_clone is None :
+                view = obi_new_view_nuc_seqs(dms.pointer, str2bytes(view_name), NULL, line_selection_p, str2bytes(comments))
+        elif not new :
+            if view_name is not None :
+                view = obi_open_view(dms.pointer, str2bytes(view_name))
+            elif view_name is None :
+                view = obi_open_view(dms.pointer, NULL)
+        
+        if view == NULL :
+            raise Exception("Error creating/opening view")
+
+        self.pointer = view
+        self.name = bytes2str(view.infos.name)
+        self.comments = bytes2str(view.infos.comments)
+        
+        # Go through columns to build list of corresponding python instances
+        self.columns = {}
+        for i in range(view.infos.column_count) :
+            column_p               = <OBIDMS_column_p> (view.columns)[i]
+            header                 = (column_p).header
+            col_name               = bytes2str(header.name)         
+            subclass               = OBIDMS_column.get_subclass_type(column_p)
+            self.columns[col_name] = subclass(self, col_name)
+        
+        self.ids = self.columns[bytes2str(ID_COLUMN)]
+        self.sequences = self.columns[bytes2str(NUC_SEQUENCE_COLUMN)]
+        self.definitions = self.columns[bytes2str(DEFINITION_COLUMN)]
+        self.qualities = self.columns[bytes2str(QUALITY_COLUMN)]
+            
+    def __getitem__(self, object item) :
+        if type(item) == str :
+            return (self.columns)[item]
+        elif type(item) == int :
+            return OBI_Nuc_Seq_Stored(self, item)
+    
+    def __setitem__(self, index_t line_idx, OBI_Nuc_Seq sequence_obj) :
+        for key in sequence_obj :
+            self[line_idx][key] = sequence_obj[key]
+            
+
+#############################################
+
+cdef class OBIView_line :
+
+    def __init__(self, OBIView view, index_t line_nb) :
+        self.index = line_nb
+        self.view = view
+    
+    def __getitem__(self, str column_name) :
+        return ((self.view).columns)[column_name][self.index]
+    
+    def __setitem__(self, str column_name, object value):
+        # TODO detect multiple elements (dict type)? put somewhere else? but more risky (in get)
+        # TODO OBI_QUAL ?
+        cdef type value_type
+        cdef str value_obitype
+        if column_name not in self.view :
+            if value == None :
+                raise Exception("Trying to create a column from a None value (can't guess type)")
+            value_type = type(value)
+            if value_type == int :
+                value_obitype = 'OBI_INT'
+            elif value_type == float :
+                value_obitype = 'OBI_FLOAT'
+            elif value_type == bool :
+                value_obitype = 'OBI_BOOL'            
+            elif value_type == str :
+                if only_ATGC(str2bytes(value)) :    # TODO detect IUPAC?
+                    value_obitype = 'OBI_SEQ'
+                elif len(value) == 1 :
+                    value_obitype = 'OBI_CHAR'
+                elif (len(value) > 1) :
+                    value_obitype = 'OBI_STR'
+            else :
+                raise Exception("Could not guess the type of a value to create a new column")
+            self.view.add_column(column_name, type=value_obitype)
+
+        (((self.view).columns)[column_name]).set_line(self.index, value)
+
+    def __contains__(self, str column_name):
+        return (column_name in self.view)
+        
+    def __repr__(self):
+        cdef dict line
+        cdef str column_name
+        line = {}
+        for column_name in self.view.columns :
+            line[column_name] = self[column_name]
+        return str(line)
+
+##########################################
 
 cdef class OBIDMS :
     
@@ -61,266 +578,127 @@ cdef class OBIDMS :
         # Fill structure and create or open the DMS
         self.dms_name = dms_name
         self.pointer = obi_dms(<const_char_p> dms_name_b)
-        # TODO: test pointer and raise Exception("Failed opening or creating an OBIDMS")
+        if self.pointer == NULL :
+            raise Exception("Failed opening or creating an OBIDMS")
 
 
     cpdef close(self) :
-        #TODO close all columns
         if (obi_close_dms(self.pointer)) < 0 :
             raise Exception("Problem closing an OBIDMS")
 
-
-    cpdef dict list(self):
     
-        # Declarations
-        cdef object p
-        cdef dict dms = {}
-        cdef str column_name
-        cdef bytes column_name_b
-        cdef str data_type
-        cdef str creation_date
-        cdef obiversion_t latest_version
-        cdef index_t line_count
-        cdef OBIDMS_column_header_p header
-        
-        p = Path(self.dms_name+'.obidms')
+    cpdef OBI_Taxonomy open_taxonomy(self, str taxo_name) :
+        return OBI_Taxonomy(self, taxo_name)
+    
 
-        print("{:<30} {:<12} {:<25} {:<30} {:<40}".format('-Column name-', 
-                                                   '-Data type-', 
-                                                   '-Latest version number-', 
-                                                   '-Line count of latest version-',
-                                                   '-Creation date of latest version-'))
-        for entry in p.iterdir():
-            if entry.suffix == ".obicol":
-                column_name = entry.stem
-                column_name_b = str2bytes(column_name)
-                dms[column_name] = {}
-                header = obi_column_get_header_from_name(self.pointer, column_name_b)
-                data_type = bytes2str(name_data_type(header.data_type))
-                line_count = header.line_count
-                creation_date = bytes2str(obi_column_format_date(header.creation_date))
-                obi_unmap_header(header)    # TODO check if error? but C will already warn and there's nothing to do
-                latest_version = obi_column_get_latest_version_from_name(self.pointer, column_name_b)
-                dms[column_name]['data_type'] = data_type
-                dms[column_name]['latest_version'] = latest_version
-                dms[column_name]['line_count'] = line_count
-                dms[column_name]['creation_date'] = creation_date
-                # TODO : actually get all the informations in the header
-                print("{:<30} {:<12} {:<25} {:<30} {:<40}".format(column_name, data_type, latest_version, line_count, creation_date))
-
-        return dms
-     
-     
-    cpdef OBIDMS_column open_column(self,
-                                    str column_name,
-                                    bint create=False, 
-                                    bint clone=False, bint clone_data=True,
-                                    obiversion_t version_number=-1, 
-                                    OBIType_t data_type= <OBIType_t> 0,
-                                    index_t nb_lines=0, 
-                                    index_t nb_elements_per_line=0, 
-                                    list elements_names=None,
-                                    str array_name="default_obiarray"):
+    cpdef OBIView open_view(self, str view_name) :
         
-        # Declarations
-        cdef OBIDMS_column column
-        cdef object subclass        # TODO object?
-        cdef bytes column_name_b
-        cdef OBIDMS_column_header_p header
+        cdef object view_class
+        cdef dict view_infos
         
-        header = NULL
+        view_infos = self.read_view_infos(view_name)
         
-        # Format the character string to send to C function
-        column_name_b = str2bytes(column_name)
-        
-        # Get the header of the latest version of the column if 
-        # some needed informations are not provided
-        if ((not data_type or not nb_elements_per_line) and not create) :
-            header = obi_column_get_header_from_name(self.pointer, column_name_b)
-            
-        # Get the data type if not provided
-        if not data_type :
-            if create :
-                raise Exception("A data type must be specified")
-            else :
-                data_type = header.data_type
-
-        # Get the number of elements per line if not provided and needed
-        if not nb_elements_per_line :
-            if create : # Set to one if not provided (default value)
-                nb_elements_per_line = 1
-            else :
-                nb_elements_per_line = header.nb_elements_per_line
-                if nb_elements_per_line > 1 :
-                    elements_names = bytes2str(header.elements_names).split(';')
-
-        if header != NULL :
-            obi_unmap_header(header)        # TODO check if error? but C will already warn and there's nothing to do
-        
-        # Open the column with the right subclass depending on the data type, the mode 
-        # (read-only or writable) and whether there are multiple elements per line or not  
-        if data_type == 1 :
-            if (create or clone) :
-                if nb_elements_per_line == 1 :
-                    subclass = OBIDMS_column_int_writable
-                else :
-                    subclass = OBIDMS_column_int_multi_elts_writable
-            else :
-                if nb_elements_per_line == 1 :
-                    subclass = OBIDMS_column_int
-                else :
-                    subclass = OBIDMS_column_int_multi_elts
-        elif data_type == 2 :
-            if (create or clone) :
-                if nb_elements_per_line == 1 :
-                    subclass = OBIDMS_column_float_writable
-                else :
-                    subclass = OBIDMS_column_float_multi_elts_writable
-            else :
-                if nb_elements_per_line == 1 :
-                    subclass = OBIDMS_column_float
-                else :
-                    subclass = OBIDMS_column_float_multi_elts
-        elif data_type == 3 :
-            if (create or clone) :
-                if nb_elements_per_line == 1 :
-                    subclass = OBIDMS_column_bool_writable
-                else :
-                    subclass = OBIDMS_column_bool_multi_elts_writable
-            else :
-                if nb_elements_per_line == 1 :
-                    subclass = OBIDMS_column_bool
-                else :
-                    subclass = OBIDMS_column_bool_multi_elts
-        elif data_type == 4 :
-            if (create or clone) :
-                if nb_elements_per_line == 1 :
-                    subclass = OBIDMS_column_char_writable
-                else :
-                    subclass = OBIDMS_column_char_multi_elts_writable
-            else :
-                if nb_elements_per_line == 1 :
-                    subclass = OBIDMS_column_char
-                else :
-                    subclass = OBIDMS_column_char_multi_elts
-        elif data_type == 5 :
-            if (create or clone) :
-                if nb_elements_per_line == 1 :
-                    subclass = OBIDMS_column_str_writable
-                else :
-                    subclass = OBIDMS_column_str_multi_elts_writable
-            else :
-                if nb_elements_per_line == 1 :
-                    subclass = OBIDMS_column_str
-                else :
-                    subclass = OBIDMS_column_str_multi_elts
+        if view_infos["view_type"] == bytes2str(VIEW_TYPE_NUC_SEQS) :
+            view_class = OBIView_NUC_SEQS
         else :
-            raise Exception("Problem with the data type")
-            
-        column = subclass(self, column_name, 
-                          create, clone, clone_data, 
-                          version_number, data_type, 
-                          nb_lines, nb_elements_per_line, 
-                          elements_names, array_name)
+            view_class = OBIView
+                
+        return view_class(self, view_name)
+    
 
-        return column
-
-
-
-cdef class OBIDMS_column :
-
-    # Should only be initialized through a subclass
-    def __init__(self, 
-                 OBIDMS dms, 
-                 str column_name,
-                 bint create, 
-                 bint clone, bint clone_data,
-                 obiversion_t version_number, 
-                 OBIType_t type,
-                 index_t nb_lines, 
-                 index_t nb_elements_per_line, 
-                 list elements_names,
-                 str array_name):
+    cpdef OBIView new_view(self, str view_name, object view_to_clone=None, list line_selection=None, str view_type=None, str comments="") :
         
-        # Declarations
-        cdef bytes column_name_b
-        cdef bytes dms_name_b
-        cdef bytes array_name_b
-        cdef bytes elements_names_b
+        cdef object view_class
         
-        # Fill structure
-        self.dms = dms
-        self.data_type = bytes2str(name_data_type(type))
-        self.column_name = column_name
-        self.nb_elements_per_line = nb_elements_per_line
-        self.elements_names = elements_names
-        
-        # Format the character strings to send them to C functions
-        column_name_b = str2bytes(column_name)
-        dms_name_b = str2bytes(self.dms.dms_name)
-        array_name_b = str2bytes(array_name)
-        
-        # Create, clone or open column
-        if create :
-            if elements_names == None :  
-                elements_names_b = column_name_b
-            else :
-                elements_names_b = str2bytes(";".join(elements_names))
-            self.pointer = obi_create_column(self.dms.pointer, column_name_b, type, 
-                                             nb_lines, nb_elements_per_line, 
-                                             elements_names_b, array_name_b)
+        if view_type is not None :
+            if view_type == bytes2str(VIEW_TYPE_NUC_SEQS) :
+                view_class = OBIView_NUC_SEQS
         else :
-            if clone :
-                self.pointer = obi_clone_column(self.dms.pointer, column_name_b, version_number, clone_data)
-            else :
-                self.pointer = obi_open_column(self.dms.pointer, column_name_b, version_number)
+            view_class = OBIView
         
-    
-    def __iter__(self):
-    
-        # Declarations
-        cdef index_t lines_used
-        cdef index_t line_nb
+        return view_class(self, view_name, new=True, view_to_clone=view_to_clone, line_selection=line_selection, comments=comments)
+
+
+    cpdef dict read_view_infos(self, str view_name) :
         
-        # Yield each line
-        lines_used = self.pointer.header.lines_used
-        for line_nb in range(lines_used):
-            yield self.get_line(line_nb)
+        cdef Obiview_infos_p      view_infos_p
+        cdef dict                 view_infos_d
+        cdef Column_reference_p   column_refs
+        cdef int                  i, j
+        cdef str                  column_name
+
+        view_infos_p = obi_view_map_file(self.pointer, str2bytes(view_name))
+        view_infos_d = {}
+        view_infos_d["name"] = bytes2str(view_infos_p.name)
+        view_infos_d["comments"] = bytes2str(view_infos_p.comments)
+        view_infos_d["view_type"] = bytes2str(view_infos_p.view_type)
+        view_infos_d["column_count"] = <int> view_infos_p.column_count
+        view_infos_d["line_count"] = <int> view_infos_p.line_count
+        view_infos_d["created_from"] = bytes2str(view_infos_p.created_from)
+        view_infos_d["creation_date"] = bytes2str(obi_format_date(view_infos_p.creation_date))
+        if (view_infos_p.all_lines) :
+            view_infos_d["line_selection"] = None
+        else :
+            view_infos_d["line_selection"] = {}
+            view_infos_d["line_selection"]["column_name"] = bytes2str((view_infos_p.line_selection).column_name)
+            view_infos_d["line_selection"]["version"] = <int> (view_infos_p.line_selection).version
+        view_infos_d["column_references"] = {}
+        column_refs = view_infos_p.column_references
+        for j in range(view_infos_d["column_count"]) :
+            column_name = bytes2str((column_refs[j]).column_name)
+            view_infos_d["column_references"][column_name] = {}
+            view_infos_d["column_references"][column_name]["version"] = column_refs[j].version
+        
+        obi_view_unmap_file(self.pointer, view_infos_p)
+
+        return view_infos_d
 
 
-    def __setitem__(self, index_t line_nb, object value):
-        self.set_line(line_nb, value)
-
-
-    def __getitem__(self, index_t line_nb):
-        return self.get_line(line_nb)
-
-
-#    cpdef object get_item(self, index_t line_nb, str element_name):           TODO
-#        raise NotImplementedError
-
-
-#    cpdef set_item(self, index_t line_nb, str element_name, object value):    TODO
-#        raise NotImplementedError
-
-    
-    cpdef list get_elements_names(self):
-        return self.elements_names
-
-             
-    cpdef str get_data_type(self):
-        return self.data_type
+#     cpdef dict read_views(self) :    # TODO function that prints the dic nicely and function that prints 1 view nicely. Add column type in col ref
+#         
+#         cdef Obiviews_infos_all_p all_views_p
+#         cdef Obiview_infos_p      view_p
+#         cdef Column_reference_p   column_refs
+#         cdef int                  nb_views
+#         cdef int                  i, j
+#         cdef str                  view_name
+#         cdef str                  column_name
+#         cdef dict                 views
+#         cdef bytes                name_b
+#         
+#         views = {}
+#         all_views_p = obi_read_view_infos(self.pointer)
+#         if all_views_p == NULL :
+#             raise Exception("No views to read")
+#         nb_views = <int> (all_views_p.header).view_count
+#         for i in range(nb_views) :
+#             view_p = (<Obiview_infos_p> (all_views_p.view_infos)) + i
+#             view_name = bytes2str(view_p.name)
+#             views[view_name] = {}
+#             views[view_name]["comments"] = bytes2str(view_p.comments)
+#             views[view_name]["view_type"] = bytes2str(view_p.view_type)
+#             views[view_name]["column_count"] = <int> view_p.column_count
+#             views[view_name]["line_count"] = <int> view_p.line_count
+#             views[view_name]["view_number"] = <int> view_p.view_number
+#             views[view_name]["created_from"] = bytes2str(view_p.created_from)
+#             views[view_name]["creation_date"] = bytes2str(obi_format_date(view_p.creation_date))
+#             if (view_p.all_lines) :
+#                 views[view_name]["line_selection"] = None
+#             else :
+#                 views[view_name]["line_selection"] = {}
+#                 views[view_name]["line_selection"]["column_name"] = bytes2str((view_p.line_selection).column_name)
+#                 views[view_name]["line_selection"]["version"] = <int> (view_p.line_selection).version
+#             views[view_name]["column_references"] = {}
+#             column_refs = view_p.column_references
+#             for j in range(views[view_name]["column_count"]) :
+#                 column_name = bytes2str((column_refs[j]).column_name)
+#                 views[view_name]["column_references"][column_name] = {}
+#                 views[view_name]["column_references"][column_name]["version"] = column_refs[j].version
+#         
+#         obi_close_view_infos(all_views_p);
+#         
+#         return views
     
     
-    cpdef index_t get_nb_lines_used(self):
-        return self.pointer.header.lines_used
     
-    
-    cpdef str get_creation_date(self):
-        return bytes2str(obi_column_format_date(self.pointer.header.creation_date))
-    
-    
-    cpdef close(self):
-        raise NotImplementedError
-   
-   
+    

python/obitools3/obidms/_obidmscolumn_bool.cfiles

@@ -1,18 +1,59 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
 ../../../src/obidmscolumn_bool.c
 ../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
 ../../../src/obidmscolumndir.c
-../../../src/obidms.h
-../../../src/obidms.c
 ../../../src/obierrno.h
 ../../../src/obierrno.c
 ../../../src/obilittlebigman.h
 ../../../src/obilittlebigman.c
 ../../../src/obitypes.h
 ../../../src/obitypes.c
-../../../src/private_at_functions.h
-../../../src/private_at_functions.c
-../../../src/obiarray.h
-../../../src/obiarray.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
+../../../src/utils.h
+../../../src/utils.c

python/obitools3/obidms/_obidmscolumn_bool.pxd

@@ -1,25 +1,14 @@
 #cython: language_level=3
 
-from .capi.obitypes cimport obibool_t, index_t
-from ._obidms       cimport OBIDMS_column
+from .capi.obitypes cimport index_t
+from ._obidms       cimport OBIDMS_column, OBIDMS_column_multi_elts
 
 
 cdef class OBIDMS_column_bool(OBIDMS_column):
     cpdef object get_line(self, index_t line_nb)
     cpdef set_line(self, index_t line_nb, object value)
-    cpdef close(self)
 
-cdef class OBIDMS_column_bool_writable(OBIDMS_column_bool):     
-    cpdef set_line(self, index_t line_nb, object value)
-    cpdef close(self)
-
-cdef class OBIDMS_column_bool_multi_elts(OBIDMS_column_bool):
+cdef class OBIDMS_column_multi_elts_bool(OBIDMS_column_multi_elts):
     cpdef object get_item(self, index_t line_nb, str element_name)
     cpdef object get_line(self, index_t line_nb)
-    cpdef set_item(self, index_t line_nb, str element_name, obibool_t value)
-    cpdef set_line(self, index_t line_nb, object values)
-
-cdef class OBIDMS_column_bool_multi_elts_writable(OBIDMS_column_bool_multi_elts):
-    cpdef set_item(self, index_t line_nb, str element_name, obibool_t value)
-    cpdef set_line(self, index_t line_nb, object values)
-    cpdef close(self)
+    cpdef set_item(self, index_t line_nb, str element_name, object value)

python/obitools3/obidms/_obidmscolumn_bool.pyx

@@ -1,13 +1,11 @@
 #cython: language_level=3
  
-from .capi.obidmscolumn cimport obi_close_column,\
-                                obi_truncate_and_close_column, \
-                                obi_column_get_obibool_with_elt_name, \
-                                obi_column_get_obibool_with_elt_idx, \
-                                obi_column_set_obibool_with_elt_name, \
-                                obi_column_set_obibool_with_elt_idx
+from .capi.obiview      cimport obi_column_get_obibool_with_elt_name_in_view, \
+                                obi_column_get_obibool_with_elt_idx_in_view, \
+                                obi_column_set_obibool_with_elt_name_in_view, \
+                                obi_column_set_obibool_with_elt_idx_in_view
 from .capi.obierrno     cimport obi_errno
-from .capi.obitypes     cimport OBIBool_NA
+from .capi.obitypes     cimport OBIBool_NA, obibool_t
 
 from obitools3.utils cimport str2bytes
 
@@ -19,7 +17,7 @@ cdef class OBIDMS_column_bool(OBIDMS_column):
     cpdef object get_line(self, index_t line_nb):
         cdef obibool_t value
         cdef object result
-        value = obi_column_get_obibool_with_elt_idx(self.pointer, line_nb, 0)
+        value = obi_column_get_obibool_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0)
         if obi_errno > 0 :
             raise IndexError(line_nb)
         if value == OBIBool_NA :
@@ -27,32 +25,20 @@ cdef class OBIDMS_column_bool(OBIDMS_column):
         else :
             result = PyBool_FromLong(value)
         return result
-
-    cpdef set_line(self, index_t line_nb, object value):
-        raise Exception("Column is read-only")
     
-    cpdef close(self):
-        if obi_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
-
-
-cdef class OBIDMS_column_bool_writable(OBIDMS_column_bool):     
-
     cpdef set_line(self, index_t line_nb, object value):
-        if obi_column_set_obibool_with_elt_idx(self.pointer, line_nb, 0, <obibool_t> value) < 0:
+        if value is None :
+            value = OBIBool_NA
+        if obi_column_set_obibool_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, <obibool_t> value) < 0:
             raise Exception("Problem setting a value in a column")
 
-    cpdef close(self):
-        if obi_truncate_and_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
 
-
-cdef class OBIDMS_column_bool_multi_elts(OBIDMS_column_bool):
+cdef class OBIDMS_column_multi_elts_bool(OBIDMS_column_multi_elts):
 
     cpdef object get_item(self, index_t line_nb, str element_name):
         cdef obibool_t value
         cdef object result
-        value = obi_column_get_obibool_with_elt_name(self.pointer, line_nb, str2bytes(element_name))
+        value = obi_column_get_obibool_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name))
         if obi_errno > 0 :
             raise IndexError(line_nb, element_name)
         if value == OBIBool_NA :
@@ -63,42 +49,29 @@ cdef class OBIDMS_column_bool_multi_elts(OBIDMS_column_bool):
     
     cpdef object get_line(self, index_t line_nb) :
         cdef obibool_t value
-        cdef object result
+        cdef object value_in_result
+        cdef dict result
         cdef index_t i
         cdef bint all_NA
         result = {}
         all_NA = True
         for i in range(self.nb_elements_per_line) :
-            value = obi_column_get_obibool_with_elt_idx(self.pointer, line_nb, i)
+            value = obi_column_get_obibool_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, i)
             if obi_errno > 0 :
-                raise IndexError(line_nb)           
-            result[self.elements_names[i]] = PyBool_FromLong(value)
-            if all_NA and (value != OBIBool_NA) :
+                raise IndexError(line_nb)
+            if value == OBIBool_NA :
+                value_in_result = None
+            else :
+                value_in_result = PyBool_FromLong(value)   
+            result[self.elements_names[i]] = value_in_result
+            if all_NA and (value_in_result is not None) :
                 all_NA = False
         if all_NA :
             result = None
         return result
 
-    cpdef set_item(self, index_t line_nb, str element_name, obibool_t value):
-        raise Exception("Column is read-only")
-
-    cpdef set_line(self, index_t line_nb, object values):
-        raise Exception("Column is read-only")
-
-
-cdef class OBIDMS_column_bool_multi_elts_writable(OBIDMS_column_bool_multi_elts):     
-
-    cpdef set_item(self, index_t line_nb, str element_name, obibool_t value):
-        if obi_column_set_obibool_with_elt_name(self.pointer, line_nb, str2bytes(element_name), value) < 0:
+    cpdef set_item(self, index_t line_nb, str element_name, object value):
+        if value is None :
+            value = OBIBool_NA
+        if obi_column_set_obibool_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), <obibool_t> value) < 0:
             raise Exception("Problem setting a value in a column")
-    
-    cpdef set_line(self, index_t line_nb, object values):
-        cdef obibool_t value
-        for element_name in values :
-            value = <obibool_t> values[element_name]
-            self.set_item(line_nb, element_name, value)
-
-    cpdef close(self):
-        if obi_truncate_and_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
-

python/obitools3/obidms/_obidmscolumn_char.cfiles

@@ -1,18 +1,59 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
 ../../../src/obidmscolumn_char.c
 ../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
 ../../../src/obidmscolumndir.c
-../../../src/obidms.h
-../../../src/obidms.c
 ../../../src/obierrno.h
 ../../../src/obierrno.c
 ../../../src/obilittlebigman.h
 ../../../src/obilittlebigman.c
 ../../../src/obitypes.h
 ../../../src/obitypes.c
-../../../src/private_at_functions.h
-../../../src/private_at_functions.c
-../../../src/obiarray.h
-../../../src/obiarray.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
+../../../src/utils.h
+../../../src/utils.c

python/obitools3/obidms/_obidmscolumn_char.pxd

@@ -1,25 +1,14 @@
 #cython: language_level=3
 
-from .capi.obitypes cimport obichar_t, index_t
-from ._obidms       cimport OBIDMS_column
+from .capi.obitypes cimport index_t
+from ._obidms       cimport OBIDMS_column, OBIDMS_column_multi_elts
 
 
 cdef class OBIDMS_column_char(OBIDMS_column):
     cpdef object get_line(self, index_t line_nb)
     cpdef set_line(self, index_t line_nb, object value)
-    cpdef close(self)
 
-cdef class OBIDMS_column_char_writable(OBIDMS_column_char):     
-    cpdef set_line(self, index_t line_nb, object value)
-    cpdef close(self)
-
-cdef class OBIDMS_column_char_multi_elts(OBIDMS_column_char):
+cdef class OBIDMS_column_multi_elts_char(OBIDMS_column_multi_elts):
     cpdef object get_item(self, index_t line_nb, str element_name)
     cpdef object get_line(self, index_t line_nb)
-    cpdef set_item(self, index_t line_nb, str element_name, bytes value)
-    cpdef set_line(self, index_t line_nb, object values)
-
-cdef class OBIDMS_column_char_multi_elts_writable(OBIDMS_column_char_multi_elts):
-    cpdef set_item(self, index_t line_nb, str element_name, bytes value)
-    cpdef set_line(self, index_t line_nb, object values)
-    cpdef close(self)
+    cpdef set_item(self, index_t line_nb, str element_name, object value)

python/obitools3/obidms/_obidmscolumn_char.pyx

@@ -1,15 +1,13 @@
 #cython: language_level=3
  
-from .capi.obidmscolumn cimport obi_close_column,\
-                                obi_truncate_and_close_column, \
-                                obi_column_get_obichar_with_elt_name, \
-                                obi_column_get_obichar_with_elt_idx, \
-                                obi_column_set_obichar_with_elt_name, \
-                                obi_column_set_obichar_with_elt_idx
+from .capi.obiview      cimport obi_column_get_obichar_with_elt_name_in_view, \
+                                obi_column_get_obichar_with_elt_idx_in_view, \
+                                obi_column_set_obichar_with_elt_name_in_view, \
+                                obi_column_set_obichar_with_elt_idx_in_view
 from .capi.obierrno     cimport obi_errno
-from .capi.obitypes     cimport OBIChar_NA
+from .capi.obitypes     cimport OBIChar_NA, obichar_t
 
-from obitools3.utils cimport str2bytes
+from obitools3.utils cimport str2bytes, bytes2str
 
 
 cdef class OBIDMS_column_char(OBIDMS_column):
@@ -17,122 +15,62 @@ cdef class OBIDMS_column_char(OBIDMS_column):
     cpdef object get_line(self, index_t line_nb):
         cdef obichar_t value
         cdef object result
-        value = obi_column_get_obichar_with_elt_idx(self.pointer, line_nb, 0)
+        value = obi_column_get_obichar_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0)
         if obi_errno > 0 :
             raise IndexError(line_nb)
         if value == OBIChar_NA :
             result = None
         else :
-            result = <bytes> value
+            result = bytes2str(value)
         return result
 
     cpdef set_line(self, index_t line_nb, object value):
-        raise Exception("Column is read-only")
-    
-    cpdef close(self):
-        if obi_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
-
-
-cdef class OBIDMS_column_char_writable(OBIDMS_column_char):     
-
-    cpdef set_line(self, index_t line_nb, object value):
-        if obi_column_set_obichar_with_elt_idx(self.pointer, line_nb, 0, <bytes> value[0]) < 0:
+        if value is None :
+            value = OBIChar_NA
+        if obi_column_set_obichar_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, str2bytes(value)[0]) < 0:
             raise Exception("Problem setting a value in a column")
 
-    cpdef close(self):
-        if obi_truncate_and_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
 
-
-cdef class OBIDMS_column_char_multi_elts(OBIDMS_column_char):
+cdef class OBIDMS_column_multi_elts_char(OBIDMS_column_multi_elts):
 
     cpdef object get_item(self, index_t line_nb, str element_name):
         cdef obichar_t value
         cdef object result
-        value = obi_column_get_obichar_with_elt_name(self.pointer, line_nb, str2bytes(element_name))
+        value = obi_column_get_obichar_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name))
         if obi_errno > 0 :
             raise IndexError(line_nb, element_name)
         if value == OBIChar_NA :
             result = None
         else :
-            result = <bytes> value
+            result = bytes2str(value)
         return result
-    
+
     cpdef object get_line(self, index_t line_nb) :
         cdef obichar_t value
-        cdef object result
+        cdef object value_in_result
+        cdef dict result
         cdef index_t i
         cdef bint all_NA
         result = {}
         all_NA = True
         for i in range(self.nb_elements_per_line) :
-            value = obi_column_get_obichar_with_elt_idx(self.pointer, line_nb, i)
+            value = obi_column_get_obichar_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, i)
             if obi_errno > 0 :
-                raise IndexError(line_nb)           
-            result[self.elements_names[i]] = <bytes> value
-            if all_NA and (value != OBIChar_NA) :
+                raise IndexError(line_nb)
+            if value == OBIChar_NA :
+                value_in_result = None
+            else :
+                value_in_result = bytes2str(value)   
+            result[self.elements_names[i]] = value_in_result
+            if all_NA and (value_in_result is not None) :
                 all_NA = False
         if all_NA :
             result = None
         return result
 
-    cpdef set_item(self, index_t line_nb, str element_name, bytes value):
-        raise Exception("Column is read-only")
-
-    cpdef set_line(self, index_t line_nb, object values):
-        raise Exception("Column is read-only")
-
-
-cdef class OBIDMS_column_char_multi_elts_writable(OBIDMS_column_char_multi_elts):     
-
-    cpdef set_item(self, index_t line_nb, str element_name, bytes value):
-        if obi_column_set_obichar_with_elt_name(self.pointer, line_nb, str2bytes(element_name), value[0]) < 0:
+    cpdef set_item(self, index_t line_nb, str element_name, object value):
+        if value is None :
+            value = OBIChar_NA
+        if obi_column_set_obichar_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), str2bytes(value)[0]) < 0:
             raise Exception("Problem setting a value in a column")
-    
-    cpdef set_line(self, index_t line_nb, object values):
-        cdef bytes value
-        for element_name in values :
-            value = <bytes> values[element_name]
-            self.set_item(line_nb, element_name, value)
 
-    cpdef close(self):
-        if obi_truncate_and_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
-
-
-
-
-
-# cdef class OBIDMS_column_char(OBIDMS_column) :
-#  
-#     cpdef object get_item(self, index_t line_nb, str element_name):
-#         cdef char value
-#         cdef object result
-#         value = obi_column_get_obichar_with_elt_name(self.pointer, line_nb, str2bytes(element_name))
-#         if obi_errno > 0 :
-#             raise IndexError(line_nb, element_name)
-#         if value == OBIChar_NA :
-#             result = None
-#         else :
-#             result = <bytes> value
-#         return result
-# 
-#     cpdef set_item(self, index_t line_nb, str element_name, bytes value):
-#         raise Exception("Column is read-only")
-#  
-#     cpdef close(self):
-#         if obi_close_column(self.pointer) < 0 :
-#             raise Exception("Problem closing a column")
-#  
-#          
-# cdef class OBIDMS_column_char_writable(OBIDMS_column_char) :     
-#  
-#     cpdef set_item(self, index_t line_nb, str element_name, bytes value):
-#         if obi_column_set_obichar_with_elt_name(self.pointer, line_nb, str2bytes(element_name), value[0]) < 0:
-#             raise Exception("Problem setting a value in a column")
-#  
-#     cpdef close(self):
-#         if obi_truncate_and_close_column(self.pointer) < 0 :
-#             raise Exception("Problem closing a column")
-#       

python/obitools3/obidms/_obidmscolumn_float.cfiles

@@ -1,18 +1,59 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
 ../../../src/obidmscolumn_float.c
 ../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
 ../../../src/obidmscolumndir.c
-../../../src/obidms.h
-../../../src/obidms.c
 ../../../src/obierrno.h
 ../../../src/obierrno.c
 ../../../src/obilittlebigman.h
 ../../../src/obilittlebigman.c
 ../../../src/obitypes.h
 ../../../src/obitypes.c
-../../../src/private_at_functions.h
-../../../src/private_at_functions.c
-../../../src/obiarray.h
-../../../src/obiarray.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
+../../../src/utils.h
+../../../src/utils.c

python/obitools3/obidms/_obidmscolumn_float.pxd

@@ -1,25 +1,14 @@
 #cython: language_level=3
 
-from .capi.obitypes cimport obifloat_t, index_t
-from ._obidms       cimport OBIDMS_column
+from .capi.obitypes cimport index_t
+from ._obidms       cimport OBIDMS_column, OBIDMS_column_multi_elts
 
 
 cdef class OBIDMS_column_float(OBIDMS_column):
     cpdef object get_line(self, index_t line_nb)
     cpdef set_line(self, index_t line_nb, object value)
-    cpdef close(self)
 
-cdef class OBIDMS_column_float_writable(OBIDMS_column_float):     
-    cpdef set_line(self, index_t line_nb, object value)
-    cpdef close(self)
-
-cdef class OBIDMS_column_float_multi_elts(OBIDMS_column_float):
+cdef class OBIDMS_column_multi_elts_float(OBIDMS_column_multi_elts):
     cpdef object get_item(self, index_t line_nb, str element_name)
     cpdef object get_line(self, index_t line_nb)
-    cpdef set_item(self, index_t line_nb, str element_name, obifloat_t value)
-    cpdef set_line(self, index_t line_nb, object values)
-
-cdef class OBIDMS_column_float_multi_elts_writable(OBIDMS_column_float_multi_elts):
-    cpdef set_item(self, index_t line_nb, str element_name, obifloat_t value)
-    cpdef set_line(self, index_t line_nb, object values)
-    cpdef close(self)
+    cpdef set_item(self, index_t line_nb, str element_name, object value)

python/obitools3/obidms/_obidmscolumn_float.pyx

@@ -1,13 +1,11 @@
 #cython: language_level=3
 
-from .capi.obidmscolumn cimport obi_close_column,\
-                                obi_truncate_and_close_column, \
-                                obi_column_get_obifloat_with_elt_name, \
-                                obi_column_get_obifloat_with_elt_idx, \
-                                obi_column_set_obifloat_with_elt_name, \
-                                obi_column_set_obifloat_with_elt_idx
+from .capi.obiview      cimport obi_column_get_obifloat_with_elt_name_in_view, \
+                                obi_column_get_obifloat_with_elt_idx_in_view, \
+                                obi_column_set_obifloat_with_elt_name_in_view, \
+                                obi_column_set_obifloat_with_elt_idx_in_view
 from .capi.obierrno     cimport obi_errno
-from .capi.obitypes     cimport OBIFloat_NA
+from .capi.obitypes     cimport OBIFloat_NA, obifloat_t
 
 from obitools3.utils cimport str2bytes
 
@@ -17,7 +15,7 @@ cdef class OBIDMS_column_float(OBIDMS_column):
     cpdef object get_line(self, index_t line_nb):
         cdef obifloat_t value
         cdef object result
-        value = obi_column_get_obifloat_with_elt_idx(self.pointer, line_nb, 0)
+        value = obi_column_get_obifloat_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0)
         if obi_errno > 0 :
             raise IndexError(line_nb)
         if value == OBIFloat_NA :
@@ -27,30 +25,18 @@ cdef class OBIDMS_column_float(OBIDMS_column):
         return result
 
     cpdef set_line(self, index_t line_nb, object value):
-        raise Exception("Column is read-only")
-    
-    cpdef close(self):
-        if obi_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
-
-
-cdef class OBIDMS_column_float_writable(OBIDMS_column_float):     
-
-    cpdef set_line(self, index_t line_nb, object value):
-        if obi_column_set_obifloat_with_elt_idx(self.pointer, line_nb, 0, <obifloat_t> value) < 0:
+        if value is None :
+            value = OBIFloat_NA
+        if obi_column_set_obifloat_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, <obifloat_t> value) < 0:
             raise Exception("Problem setting a value in a column")
 
-    cpdef close(self):
-        if obi_truncate_and_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
 
-
-cdef class OBIDMS_column_float_multi_elts(OBIDMS_column_float):
+cdef class OBIDMS_column_multi_elts_float(OBIDMS_column_multi_elts):
 
     cpdef object get_item(self, index_t line_nb, str element_name):
         cdef obifloat_t value
         cdef object result
-        value = obi_column_get_obifloat_with_elt_name(self.pointer, line_nb, str2bytes(element_name))
+        value = obi_column_get_obifloat_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name))
         if obi_errno > 0 :
             raise IndexError(line_nb, element_name)
         if value == OBIFloat_NA :
@@ -58,46 +44,33 @@ cdef class OBIDMS_column_float_multi_elts(OBIDMS_column_float):
         else :
             result = <double> value
         return result
-    
+
     cpdef object get_line(self, index_t line_nb) :
         cdef obifloat_t value
-        cdef object result
+        cdef object value_in_result
+        cdef dict result
         cdef index_t i
         cdef bint all_NA
         result = {}
         all_NA = True
         for i in range(self.nb_elements_per_line) :
-            value = obi_column_get_obifloat_with_elt_idx(self.pointer, line_nb, i)
+            value = obi_column_get_obifloat_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, i)
             if obi_errno > 0 :
-                raise IndexError(line_nb)           
-            result[self.elements_names[i]] = <double> value
-            if all_NA and (value != OBIFloat_NA) :
+                raise IndexError(line_nb)
+            if value == OBIFloat_NA :
+                value_in_result = None
+            else :
+                value_in_result = <double> value
+            result[self.elements_names[i]] = value_in_result
+            if all_NA and (value_in_result is not None) :
                 all_NA = False
         if all_NA :
             result = None
         return result
-
-    cpdef set_item(self, index_t line_nb, str element_name, obifloat_t value):
-        raise Exception("Column is read-only")
-
-    cpdef set_line(self, index_t line_nb, object values):
-        raise Exception("Column is read-only")
-
-
-cdef class OBIDMS_column_float_multi_elts_writable(OBIDMS_column_float_multi_elts):     
-
-    cpdef set_item(self, index_t line_nb, str element_name, obifloat_t value):
-        if obi_column_set_obifloat_with_elt_name(self.pointer, line_nb, str2bytes(element_name), value) < 0:
+   
+    cpdef set_item(self, index_t line_nb, str element_name, object value):
+        if value is None :
+            value = OBIFloat_NA
+        if obi_column_set_obifloat_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), <obifloat_t> value) < 0:
             raise Exception("Problem setting a value in a column")
-    
-    cpdef set_line(self, index_t line_nb, object values):
-        cdef obifloat_t value
-        for element_name in values :
-            value = <obifloat_t> values[element_name]
-            self.set_item(line_nb, element_name, value)
-
-    cpdef close(self):
-        if obi_truncate_and_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
-
  

python/obitools3/obidms/_obidmscolumn_int.cfiles

@@ -1,18 +1,59 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
 ../../../src/obidmscolumn_int.c
 ../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
 ../../../src/obidmscolumndir.c
-../../../src/obidms.h
-../../../src/obidms.c
 ../../../src/obierrno.h
 ../../../src/obierrno.c
 ../../../src/obilittlebigman.h
 ../../../src/obilittlebigman.c
 ../../../src/obitypes.h
 ../../../src/obitypes.c
-../../../src/private_at_functions.h
-../../../src/private_at_functions.c
-../../../src/obiarray.h
-../../../src/obiarray.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
+../../../src/utils.h
+../../../src/utils.c

python/obitools3/obidms/_obidmscolumn_int.pxd

@@ -1,25 +1,14 @@
 #cython: language_level=3
 
-from .capi.obitypes cimport obiint_t, index_t
-from ._obidms       cimport OBIDMS_column
+from .capi.obitypes cimport index_t
+from ._obidms       cimport OBIDMS_column, OBIDMS_column_multi_elts
 
 
 cdef class OBIDMS_column_int(OBIDMS_column):
     cpdef object get_line(self, index_t line_nb)
     cpdef set_line(self, index_t line_nb, object value)
-    cpdef close(self)
 
-cdef class OBIDMS_column_int_writable(OBIDMS_column_int):     
-    cpdef set_line(self, index_t line_nb, object value)
-    cpdef close(self)
-
-cdef class OBIDMS_column_int_multi_elts(OBIDMS_column_int):
+cdef class OBIDMS_column_multi_elts_int(OBIDMS_column_multi_elts):
     cpdef object get_item(self, index_t line_nb, str element_name)
     cpdef object get_line(self, index_t line_nb)
-    cpdef set_item(self, index_t line_nb, str element_name, obiint_t value)
-    cpdef set_line(self, index_t line_nb, object values)
-
-cdef class OBIDMS_column_int_multi_elts_writable(OBIDMS_column_int_multi_elts):
-    cpdef set_item(self, index_t line_nb, str element_name, obiint_t value)
-    cpdef set_line(self, index_t line_nb, object values)
-    cpdef close(self)
+    cpdef set_item(self, index_t line_nb, str element_name, object value)

python/obitools3/obidms/_obidmscolumn_int.pyx

@@ -1,13 +1,11 @@
 #cython: language_level=3
 
-from .capi.obidmscolumn cimport obi_close_column,\
-                                obi_truncate_and_close_column, \
-                                obi_column_get_obiint_with_elt_name, \
-                                obi_column_get_obiint_with_elt_idx, \
-                                obi_column_set_obiint_with_elt_name, \
-                                obi_column_set_obiint_with_elt_idx
+from .capi.obiview      cimport obi_column_get_obiint_with_elt_name_in_view, \
+                                obi_column_get_obiint_with_elt_idx_in_view, \
+                                obi_column_set_obiint_with_elt_name_in_view, \
+                                obi_column_set_obiint_with_elt_idx_in_view
 from .capi.obierrno     cimport obi_errno
-from .capi.obitypes     cimport OBIInt_NA
+from .capi.obitypes     cimport OBIInt_NA, obiint_t
 
 from obitools3.utils cimport str2bytes
 
@@ -19,7 +17,7 @@ cdef class OBIDMS_column_int(OBIDMS_column):
     cpdef object get_line(self, index_t line_nb):
         cdef obiint_t value
         cdef object result
-        value = obi_column_get_obiint_with_elt_idx(self.pointer, line_nb, 0)
+        value = obi_column_get_obiint_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0)
         if obi_errno > 0 :
             raise IndexError(line_nb)
         if value == OBIInt_NA :
@@ -27,32 +25,20 @@ cdef class OBIDMS_column_int(OBIDMS_column):
         else :
             result = PyInt_FromLong(value)
         return result
-
+            
     cpdef set_line(self, index_t line_nb, object value):
-        raise Exception("Column is read-only")
-    
-    cpdef close(self):
-        if obi_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
-
-
-cdef class OBIDMS_column_int_writable(OBIDMS_column_int):     
-
-    cpdef set_line(self, index_t line_nb, object value):
-        if obi_column_set_obiint_with_elt_idx(self.pointer, line_nb, 0, <obiint_t> value) < 0:
+        if value is None :
+            value = OBIInt_NA
+        if obi_column_set_obiint_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, <obiint_t> value) < 0:
             raise Exception("Problem setting a value in a column")
 
-    cpdef close(self):
-        if obi_truncate_and_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
 
-
-cdef class OBIDMS_column_int_multi_elts(OBIDMS_column_int):
+cdef class OBIDMS_column_multi_elts_int(OBIDMS_column_multi_elts):
 
     cpdef object get_item(self, index_t line_nb, str element_name):
         cdef obiint_t value
         cdef object result
-        value = obi_column_get_obiint_with_elt_name(self.pointer, line_nb, str2bytes(element_name))
+        value = obi_column_get_obiint_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name))
         if obi_errno > 0 :
             raise IndexError(line_nb, element_name)
         if value == OBIInt_NA :
@@ -63,42 +49,30 @@ cdef class OBIDMS_column_int_multi_elts(OBIDMS_column_int):
     
     cpdef object get_line(self, index_t line_nb) :
         cdef obiint_t value
-        cdef object result
+        cdef object value_in_result
+        cdef dict result
         cdef index_t i
         cdef bint all_NA
         result = {}
         all_NA = True
         for i in range(self.nb_elements_per_line) :
-            value = obi_column_get_obiint_with_elt_idx(self.pointer, line_nb, i)
+            value = obi_column_get_obiint_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, i)
             if obi_errno > 0 :
-                raise IndexError(line_nb)           
-            result[self.elements_names[i]] = PyInt_FromLong(value)
-            if all_NA and (value != OBIInt_NA) :
+                raise IndexError(line_nb)
+            if value == OBIInt_NA :
+                value_in_result = None
+            else :
+                value_in_result = PyInt_FromLong(value)
+            result[self.elements_names[i]] = value_in_result
+            if all_NA and (value_in_result is not None) :
                 all_NA = False
         if all_NA :
-            result = None
+            result = None   # TODO discuss
         return result
 
-    cpdef set_item(self, index_t line_nb, str element_name, obiint_t value):
-        raise Exception("Column is read-only")
-
-    cpdef set_line(self, index_t line_nb, object values):
-        raise Exception("Column is read-only")
-
-
-cdef class OBIDMS_column_int_multi_elts_writable(OBIDMS_column_int_multi_elts):     
-
-    cpdef set_item(self, index_t line_nb, str element_name, obiint_t value):
-        if obi_column_set_obiint_with_elt_name(self.pointer, line_nb, str2bytes(element_name), value) < 0:
+    cpdef set_item(self, index_t line_nb, str element_name, object value):
+        if value is None :
+            value = OBIInt_NA
+        if obi_column_set_obiint_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), <obiint_t> value) < 0:
             raise Exception("Problem setting a value in a column")
-    
-    cpdef set_line(self, index_t line_nb, object values):
-        cdef obiint_t value
-        for element_name in values :
-            value = <obiint_t> values[element_name]
-            self.set_item(line_nb, element_name, value)
-
-    cpdef close(self):
-        if obi_truncate_and_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
 

python/obitools3/obidms/_obidmscolumn_qual.cfiles

@@ -0,0 +1,59 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
+../../../src/obidmscolumn.h
+../../../src/obidmscolumn.c
+../../../src/obidmscolumndir.h
+../../../src/obidmscolumndir.c
+../../../src/obierrno.h
+../../../src/obierrno.c
+../../../src/obilittlebigman.h
+../../../src/obilittlebigman.c
+../../../src/obitypes.h
+../../../src/obitypes.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
+../../../src/utils.h
+../../../src/utils.c

python/obitools3/obidms/_obidmscolumn_qual.pxd

@@ -0,0 +1,20 @@
+#cython: language_level=3
+
+from .capi.obitypes cimport index_t
+from ._obidms       cimport OBIDMS_column , OBIDMS_column_multi_elts
+
+
+cdef class OBIDMS_column_qual(OBIDMS_column):
+    cpdef object get_line(self, index_t line_nb)
+    cpdef object get_str_line(self, index_t line_nb)
+    cpdef set_line(self, index_t line_nb, object value)
+    cpdef set_str_line(self, index_t line_nb, object value)
+
+
+cdef class OBIDMS_column_multi_elts_qual(OBIDMS_column_multi_elts):
+    cpdef object get_item(self, index_t line_nb, str element_name)
+    cpdef object get_str_item(self, index_t line_nb, str element_name)
+    cpdef object get_line(self, index_t line_nb)
+    cpdef object get_str_line(self, index_t line_nb)
+    cpdef set_item(self, index_t line_nb, str element_name, object value)
+    cpdef set_str_item(self, index_t line_nb, str element_name, object value)

python/obitools3/obidms/_obidmscolumn_qual.pyx

@@ -0,0 +1,184 @@
+#cython: language_level=3
+ 
+from .capi.obiview      cimport obi_column_get_obiqual_char_with_elt_name_in_view, \
+                                obi_column_get_obiqual_char_with_elt_idx_in_view, \
+                                obi_column_set_obiqual_char_with_elt_name_in_view, \
+                                obi_column_set_obiqual_char_with_elt_idx_in_view, \
+                                obi_column_get_obiqual_int_with_elt_name_in_view, \
+                                obi_column_get_obiqual_int_with_elt_idx_in_view, \
+                                obi_column_set_obiqual_int_with_elt_name_in_view, \
+                                obi_column_set_obiqual_int_with_elt_idx_in_view
+                                
+from .capi.obierrno     cimport obi_errno
+from .capi.obitypes     cimport OBIQual_char_NA, OBIQual_int_NA, const_char_p
+
+from ._obidms           cimport OBIView
+
+from obitools3.utils cimport str2bytes, bytes2str
+
+from libc.stdlib cimport free
+from libc.string cimport strcmp
+from libc.stdint cimport uint8_t
+from libc.stdlib cimport malloc
+
+
+cdef class OBIDMS_column_qual(OBIDMS_column):
+
+    cpdef object get_line(self, index_t line_nb):
+        cdef const uint8_t*  value
+        cdef int             value_length
+        cdef object          result
+        cdef int             i
+        value = obi_column_get_obiqual_int_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, &value_length)
+        if obi_errno > 0 :
+            raise IndexError(line_nb)
+        if value == OBIQual_int_NA :
+            result = None
+        else :
+            result = []
+            for i in range(value_length) :
+                result.append(<int>value[i])            
+        return result
+    
+    cpdef object get_str_line(self, index_t line_nb):
+        cdef char*  value
+        cdef object result
+        cdef int    i
+        value = obi_column_get_obiqual_char_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0)
+        if obi_errno > 0 :
+            raise IndexError(line_nb)
+        if value == OBIQual_char_NA :
+            result = None
+        else :
+            result = bytes2str(value)
+            free(value)
+        return result
+
+    cpdef set_line(self, index_t line_nb, object value):
+        cdef uint8_t* value_b
+        cdef int      value_length
+        if value is None :
+            if obi_column_set_obiqual_int_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, OBIQual_int_NA, 0) < 0:
+                raise Exception("Problem setting a value in a column")
+        else :
+            value_length = len(value)
+            value_b = <uint8_t*> malloc(value_length * sizeof(uint8_t))
+            for i in range(value_length) :
+                value_b[i] = <uint8_t>value[i]
+            if obi_column_set_obiqual_int_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, value_b, value_length) < 0:
+                raise Exception("Problem setting a value in a column")
+            free(value_b)
+
+    cpdef set_str_line(self, index_t line_nb, object value):
+        if value is None :
+            if obi_column_set_obiqual_char_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, OBIQual_char_NA) < 0:
+                raise Exception("Problem setting a value in a column")
+        else :
+            if obi_column_set_obiqual_char_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, str2bytes(value)) < 0:
+                raise Exception("Problem setting a value in a column")
+
+
+cdef class OBIDMS_column_multi_elts_qual(OBIDMS_column_multi_elts):
+ 
+    cpdef object get_item(self, index_t line_nb, str element_name):
+        cdef const uint8_t*  value
+        cdef int             value_length
+        cdef object          result
+        cdef int             i
+        value = obi_column_get_obiqual_int_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), &value_length)
+        if obi_errno > 0 :
+            raise IndexError(line_nb, element_name)
+        if value == OBIQual_int_NA :
+            result = None
+        else :
+            result = []
+            for i in range(value_length) :
+                result.append(<int>value[i])            
+        return result
+ 
+    cpdef object get_str_item(self, index_t line_nb, str element_name):
+        cdef char*  value
+        cdef object result
+        value = obi_column_get_obiqual_char_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name))
+        if obi_errno > 0 :
+            raise IndexError(line_nb, element_name)
+        if value == OBIQual_char_NA :
+            result = None
+        else :
+            result = bytes2str(value)
+            free(value)
+        return result
+
+    cpdef object get_line(self, index_t line_nb) :
+        cdef const uint8_t*  value
+        cdef int             value_length
+        cdef object          value_in_result
+        cdef dict            result
+        cdef index_t         i
+        cdef int             j
+        cdef bint            all_NA
+        result = {}
+        all_NA = True
+        for i in range(self.nb_elements_per_line) :
+            value = obi_column_get_obiqual_int_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, i, &value_length)
+            if obi_errno > 0 :
+                raise IndexError(line_nb)
+            if value == OBIQual_int_NA :
+                value_in_result = None
+            else :
+                value_in_result = []
+                for j in range(value_length) :
+                    value_in_result.append(<int>value[j])            
+            result[self.elements_names[i]] = value_in_result
+            if all_NA and (value_in_result is not None) :
+                all_NA = False
+        if all_NA :
+            result = None
+        return result
+ 
+    cpdef object get_str_line(self, index_t line_nb) :
+        cdef char*   value
+        cdef object  value_in_result
+        cdef dict    result
+        cdef index_t i
+        cdef bint    all_NA
+        result = {}
+        all_NA = True
+        for i in range(self.nb_elements_per_line) :
+            value = obi_column_get_obiqual_char_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, i)
+            if obi_errno > 0 :
+                raise IndexError(line_nb)
+            if value == OBIQual_char_NA :
+                value_in_result = None
+            else :
+                value_in_result = bytes2str(value)
+                free(value)
+            result[self.elements_names[i]] = value_in_result
+            if all_NA and (value_in_result is not None) :
+                all_NA = False
+        if all_NA :
+            result = None
+        return result
+
+    cpdef set_item(self, index_t line_nb, str element_name, object value):
+        cdef uint8_t* value_b
+        cdef int      value_length
+        if value is None :
+            if obi_column_set_obiqual_int_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), OBIQual_int_NA, 0) < 0:
+                raise Exception("Problem setting a value in a column")
+        else :
+            value_length = len(value)
+            value_b = <uint8_t*> malloc(value_length * sizeof(uint8_t))
+            for i in range(value_length) :
+                value_b[i] = <uint8_t>value[i]
+            if obi_column_set_obiqual_int_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), value_b, value_length) < 0:
+                raise Exception("Problem setting a value in a column")
+            free(value_b)
+
+    cpdef set_str_item(self, index_t line_nb, str element_name, object value):
+        if value is None :
+            if obi_column_set_obiqual_char_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), OBIQual_char_NA) < 0:
+                raise Exception("Problem setting a value in a column")
+        else :
+            if obi_column_set_obiqual_char_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), str2bytes(value)) < 0:
+                raise Exception("Problem setting a value in a column")

python/obitools3/obidms/_obidmscolumn_seq.cfiles

@@ -0,0 +1,59 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
+../../../src/obidmscolumn.h
+../../../src/obidmscolumn.c
+../../../src/obidmscolumndir.h
+../../../src/obidmscolumndir.c
+../../../src/obierrno.h
+../../../src/obierrno.c
+../../../src/obilittlebigman.h
+../../../src/obilittlebigman.c
+../../../src/obitypes.h
+../../../src/obitypes.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
+../../../src/utils.h
+../../../src/utils.c

python/obitools3/obidms/_obidmscolumn_seq.pxd

@@ -0,0 +1,26 @@
+#cython: language_level=3
+
+from .capi.obitypes cimport index_t
+from ._obidms       cimport OBIView, OBIDMS_column, OBIDMS_column_multi_elts
+
+
+cdef class OBIDMS_column_seq(OBIDMS_column):
+    cpdef object get_line(self, index_t line_nb)
+    cpdef set_line(self, index_t line_nb, object value)
+    
+    # TO DISCUSS : 
+    # I'am not sure that this method has to be declared here
+    # Alignment must be declared outside of the sequence object
+    cpdef align(self, 
+                OBIView score_view,
+                OBIDMS_column score_column, 
+                double threshold = *, 
+                bint normalize = *, 
+                int reference = *,
+                bint similarity_mode = *)
+
+
+cdef class OBIDMS_column_multi_elts_seq(OBIDMS_column_multi_elts):
+    cpdef object get_item(self, index_t line_nb, str element_name)
+    cpdef object get_line(self, index_t line_nb)
+    cpdef set_item(self, index_t line_nb, str element_name, object value)

python/obitools3/obidms/_obidmscolumn_seq.pyx

@@ -0,0 +1,127 @@
+#cython: language_level=3
+ 
+from .capi.obiview      cimport obi_column_get_obiseq_with_elt_name_in_view, \
+                                obi_column_get_obiseq_with_elt_idx_in_view, \
+                                obi_column_set_obiseq_with_elt_name_in_view, \
+                                obi_column_set_obiseq_with_elt_idx_in_view
+from .capi.obialign     cimport obi_align_one_column
+from .capi.obierrno     cimport obi_errno
+from .capi.obitypes     cimport OBISeq_NA, const_char_p
+
+from ._obidms           cimport OBIView
+
+from obitools3.utils cimport str2bytes, bytes2str
+
+from libc.stdlib cimport free
+
+
+cdef class OBIDMS_column_seq(OBIDMS_column):
+
+    cpdef object get_line(self, index_t line_nb):
+        cdef char*  value
+        cdef object result
+        value = obi_column_get_obiseq_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0)
+        if obi_errno > 0 :
+            raise IndexError(line_nb)
+        if value == OBISeq_NA :
+            result = None
+        else :
+            try:
+                result = <bytes> value
+            finally:
+                free(value)
+        return result
+
+    cpdef set_line(self, index_t line_nb, object value):
+        cdef bytes value_b
+
+        if value is None :
+            value_b = OBISeq_NA
+        elif isinstance(value, bytes) :
+            value_b = value
+        elif isinstance(value, str) :
+            value_b = str2bytes(value)
+        else:
+            raise TypeError('Sequence value must be of type Bytes, Str or None')
+
+        if obi_column_set_obiseq_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, value_b) < 0:
+            raise Exception("Problem setting a value in a column")
+        else :
+            if obi_column_set_obiseq_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, str2bytes(value)) < 0:
+                raise Exception("Problem setting a value in a column")
+    
+    # TODO choose alignment type (lcs or other) with supplementary argument
+    cpdef align(self, 
+                OBIView score_view,
+                OBIDMS_column score_column, 
+                double threshold = 0.0, 
+                bint normalize = True, 
+                int reference = 0,      # TODO
+                bint similarity_mode = True):
+        if (obi_align_one_column(self.view.pointer, (self.pointer)[0], score_view.pointer, (score_column.pointer)[0], threshold, normalize, reference, similarity_mode) < 0) :
+            raise Exception("An error occurred while aligning sequences")
+
+        
+
+cdef class OBIDMS_column_multi_elts_seq(OBIDMS_column_multi_elts):
+
+    cpdef object get_item(self, index_t line_nb, str element_name):
+        cdef char*  value
+        cdef object result
+        value = obi_column_get_obiseq_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name))
+        if obi_errno > 0 :
+            raise IndexError(line_nb, element_name)
+        if value == OBISeq_NA :
+            result = None
+        else :
+            try:
+                result = <bytes> value
+            finally:
+                free(value)
+        return result
+
+    
+    cpdef object get_line(self, index_t line_nb) :
+        cdef char*   value
+        cdef object  value_in_result
+        cdef dict    result
+        cdef index_t i
+        cdef bint    all_NA
+        result = {}
+        all_NA = True
+        for i in range(self.nb_elements_per_line) :
+            value = obi_column_get_obiseq_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, i)
+            if obi_errno > 0 :
+                raise IndexError(line_nb)
+            if value == OBISeq_NA :
+                value_in_result = None
+            else :
+                try:
+                    value_in_result = <bytes> value
+                finally:
+                    free(value)    
+            result[self.elements_names[i]] = value_in_result
+            if all_NA and (value_in_result is not None) :
+                all_NA = False
+        if all_NA :
+            result = None
+        return result
+
+    cpdef set_item(self, index_t line_nb, str element_name, object value):
+        cdef bytes value_b
+        
+        if value is None :
+            value_b = OBISeq_NA
+        elif isinstance(value, bytes) :
+            value_b = value
+        elif isinstance(value, str) :
+            value_b = str2bytes(value)
+        else:
+            raise TypeError('Sequence value must be of type Bytes, Str or None')
+        
+        if obi_column_set_obiseq_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), value_b) < 0:
+            raise Exception("Problem setting a value in a column")
+    
+#    cpdef align(self, ):        # TODO
+#        raise Exception("Columns with multiple sequences per line can't be aligned")    # TODO discuss
+

python/obitools3/obidms/_obidmscolumn_str.cfiles

@@ -1,18 +1,59 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
 ../../../src/obidmscolumn_str.c
 ../../../src/obidmscolumn_str.h
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
 ../../../src/obidmscolumndir.c
-../../../src/obidms.h
-../../../src/obidms.c
 ../../../src/obierrno.h
 ../../../src/obierrno.c
 ../../../src/obilittlebigman.h
 ../../../src/obilittlebigman.c
 ../../../src/obitypes.h
 ../../../src/obitypes.c
-../../../src/private_at_functions.h
-../../../src/private_at_functions.c
-../../../src/obiarray.h
-../../../src/obiarray.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
+../../../src/utils.h
+../../../src/utils.c

python/obitools3/obidms/_obidmscolumn_str.pxd

@@ -1,25 +1,14 @@
 #cython: language_level=3
 
 from .capi.obitypes cimport index_t
-from ._obidms       cimport OBIDMS_column
+from ._obidms       cimport OBIDMS_column, OBIDMS_column_multi_elts
 
 
 cdef class OBIDMS_column_str(OBIDMS_column):
     cpdef object get_line(self, index_t line_nb)
     cpdef set_line(self, index_t line_nb, object value)
-    cpdef close(self)
 
-cdef class OBIDMS_column_str_writable(OBIDMS_column_str):     
-    cpdef set_line(self, index_t line_nb, object value)
-    cpdef close(self)
-
-cdef class OBIDMS_column_str_multi_elts(OBIDMS_column_str):
+cdef class OBIDMS_column_multi_elts_str(OBIDMS_column_multi_elts):
     cpdef object get_item(self, index_t line_nb, str element_name)
     cpdef object get_line(self, index_t line_nb)
-    cpdef set_item(self, index_t line_nb, str element_name, str value)
-    cpdef set_line(self, index_t line_nb, object values)
-
-cdef class OBIDMS_column_str_multi_elts_writable(OBIDMS_column_str_multi_elts):
-    cpdef set_item(self, index_t line_nb, str element_name, str value)
-    cpdef set_line(self, index_t line_nb, object values)
-    cpdef close(self)
+    cpdef set_item(self, index_t line_nb, str element_name, object value)

python/obitools3/obidms/_obidmscolumn_str.pyx

@@ -1,13 +1,11 @@
 #cython: language_level=3
  
-from .capi.obidmscolumn cimport obi_close_column,\
-                                obi_truncate_and_close_column, \
-                                obi_column_get_obistr_with_elt_name, \
-                                obi_column_get_obistr_with_elt_idx, \
-                                obi_column_set_obistr_with_elt_name, \
-                                obi_column_set_obistr_with_elt_idx
+from .capi.obiview      cimport obi_column_get_obistr_with_elt_name_in_view, \
+                                obi_column_get_obistr_with_elt_idx_in_view, \
+                                obi_column_set_obistr_with_elt_name_in_view, \
+                                obi_column_set_obistr_with_elt_idx_in_view
 from .capi.obierrno     cimport obi_errno
-from .capi.obitypes     cimport OBIIdx_NA, const_char_p
+from .capi.obitypes     cimport OBIStr_NA, const_char_p
 
 from obitools3.utils cimport str2bytes, bytes2str
 
@@ -15,89 +13,72 @@ from obitools3.utils cimport str2bytes, bytes2str
 cdef class OBIDMS_column_str(OBIDMS_column):
 
     cpdef object get_line(self, index_t line_nb):
-        cdef bytes value
-        cdef object result
-        value = <bytes> obi_column_get_obistr_with_elt_idx(self.pointer, line_nb, 0)
+        cdef const_char_p value
+        cdef object       result
+        value = obi_column_get_obistr_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0)
         if obi_errno > 0 :
             raise IndexError(line_nb)
-        if value == OBIIdx_NA :
+        if value == OBIStr_NA :
             result = None
         else :
             result = bytes2str(value)
+        # NOTE: value is not freed because the pointer points to a mmapped region in an AVL data file. (TODO discuss)
         return result
 
     cpdef set_line(self, index_t line_nb, object value):
-        raise Exception("Column is read-only")
-    
-    cpdef close(self):
-        if obi_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
+        if value is None :
+            if obi_column_set_obistr_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, OBIStr_NA) < 0:
+                raise Exception("Problem setting a value in a column")
+        else :
+            if obi_column_set_obistr_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, str2bytes(value)) < 0:
+                raise Exception("Problem setting a value in a column")
 
 
-cdef class OBIDMS_column_str_writable(OBIDMS_column_str):     
-
-    cpdef set_line(self, index_t line_nb, object value):
-        if obi_column_set_obistr_with_elt_idx(self.pointer, line_nb, 0, str2bytes(value)) < 0:
-            raise Exception("Problem setting a value in a column")
-
-    cpdef close(self):
-        if obi_truncate_and_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
-
-
-cdef class OBIDMS_column_str_multi_elts(OBIDMS_column_str):
+cdef class OBIDMS_column_multi_elts_str(OBIDMS_column_multi_elts):
 
     cpdef object get_item(self, index_t line_nb, str element_name):
-        cdef bytes value
-        cdef object result
-        value = <bytes> obi_column_get_obistr_with_elt_name(self.pointer, line_nb, str2bytes(element_name))
+        cdef const_char_p value
+        cdef object       result
+        value = obi_column_get_obistr_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name))
         if obi_errno > 0 :
             raise IndexError(line_nb, element_name)
-        if value == OBIIdx_NA :
+        if value == OBIStr_NA :
             result = None
         else :
             result = bytes2str(value)
+        # NOTE: value is not freed because the pointer points to a mmapped region in an AVL data file. (TODO discuss)
         return result
-    
+
     cpdef object get_line(self, index_t line_nb) :
-        cdef bytes value
-        cdef object result
-        cdef index_t i
-        cdef bint all_NA
+        cdef const_char_p value
+        cdef object       value_in_result
+        cdef dict         result
+        cdef index_t      i
+        cdef bint         all_NA
         result = {}
         all_NA = True
         for i in range(self.nb_elements_per_line) :
-            value = <bytes> obi_column_get_obistr_with_elt_idx(self.pointer, line_nb, i)
+            value = obi_column_get_obistr_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, i)
             if obi_errno > 0 :
-                raise IndexError(line_nb)           
-            result[self.elements_names[i]] = bytes2str(value)
-            if all_NA and (value != OBIIdx_NA) :
+                raise IndexError(line_nb)
+            if value == OBIStr_NA :
+                value_in_result = None
+            else :
+                value_in_result = bytes2str(value)
+            # NOTE: value is not freed because the pointer points to a mmapped region in an AVL data file. (TODO discuss)
+            result[self.elements_names[i]] = value_in_result
+            if all_NA and (value_in_result is not None) :
                 all_NA = False
         if all_NA :
             result = None
         return result
 
-    cpdef set_item(self, index_t line_nb, str element_name, str value):
-        raise Exception("Column is read-only")
-
-    cpdef set_line(self, index_t line_nb, object values):
-        raise Exception("Column is read-only")
-
-
-cdef class OBIDMS_column_str_multi_elts_writable(OBIDMS_column_str_multi_elts):     
-
-    cpdef set_item(self, index_t line_nb, str element_name, str value):
-        if obi_column_set_obistr_with_elt_name(self.pointer, line_nb, str2bytes(element_name), str2bytes(value)) < 0:
+    cpdef set_item(self, index_t line_nb, str element_name, object value):
+        cdef bytes value_b
+        if value is None :
+            value_b = OBIStr_NA
+        else :
+            value_b = str2bytes(value)
+        if obi_column_set_obistr_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), value_b) < 0:
             raise Exception("Problem setting a value in a column")
     
-    cpdef set_line(self, index_t line_nb, object values):
-        cdef str value
-        for element_name in values :
-            value = values[element_name]
-            self.set_item(line_nb, element_name, value)
-
-    cpdef close(self):
-        if obi_truncate_and_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
-
- 

python/obitools3/obidms/_obiseq.cfiles

@@ -0,0 +1,59 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
+../../../src/obidmscolumn.h
+../../../src/obidmscolumn.c
+../../../src/obidmscolumndir.h
+../../../src/obidmscolumndir.c
+../../../src/obierrno.h
+../../../src/obierrno.c
+../../../src/obilittlebigman.h
+../../../src/obilittlebigman.c
+../../../src/obitypes.h
+../../../src/obitypes.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
+../../../src/utils.h
+../../../src/utils.c

python/obitools3/obidms/_obiseq.pxd

@@ -0,0 +1,38 @@
+#cython: language_level=3
+
+from ._obidms cimport OBIView_line
+
+
+cdef class OBI_Seq(dict) :
+    cdef object id
+    cdef object definition
+    cdef object sequence
+
+    cpdef set_id(self, object id)
+    cpdef object get_id(self)
+    cpdef set_definition(self, object definition)
+    cpdef object get_definition(self)
+    cpdef object get_sequence(self)
+
+
+cdef class OBI_Nuc_Seq(OBI_Seq) :
+    cdef object quality
+    
+    #cpdef object reverse_complement(self)    
+    cpdef set_sequence(self, object sequence)
+    cpdef set_quality(self, object quality)
+    cpdef object get_quality(self)
+
+
+cdef class OBI_Nuc_Seq_Stored(OBIView_line) :
+    cpdef set_id(self, object id)
+    cpdef object get_id(self)
+    cpdef set_definition(self, object definition)
+    cpdef object get_definition(self)
+    cpdef set_sequence(self, object sequence)
+    cpdef object get_sequence(self)
+    cpdef set_quality(self, object quality)
+    cpdef object get_quality(self)
+    cpdef object get_str_quality(self)
+
+   # cpdef object reverse_complement(self)

python/obitools3/obidms/_obiseq.pyx

@@ -0,0 +1,97 @@
+#cython: language_level=3
+
+from obitools3.utils cimport bytes2str, str2bytes
+
+from .capi.obiview cimport NUC_SEQUENCE_COLUMN, \
+                           ID_COLUMN, \
+                           DEFINITION_COLUMN, \
+                           QUALITY_COLUMN
+
+cdef class OBI_Seq(dict) :
+
+    def __init__(self, object id, object seq, object definition=None) :
+        self.set_id(id)
+        self.set_sequence(seq)
+        if definition is not None :
+            self.set_definition(definition)
+        
+    cpdef set_id(self, object id) :
+        self.id = id
+        self[bytes2str(ID_COLUMN)] = id
+        
+    cpdef get_id(self) :
+        return self.id
+            
+    cpdef set_definition(self, object definition) :
+        self.definition = definition
+        self[bytes2str(DEFINITION_COLUMN)] = definition
+        
+    cpdef get_definition(self) :
+        return self.definition
+
+    cpdef get_sequence(self) :
+        return self.sequence
+
+    def __str__(self) :
+        return self.sequence    # or not
+
+
+cdef class OBI_Nuc_Seq(OBI_Seq) :
+    
+    cpdef set_sequence(self, object sequence) :
+        self.sequence = sequence
+        self[bytes2str(NUC_SEQUENCE_COLUMN)] = sequence
+
+    cpdef set_quality(self, object quality) :
+        self.quality = quality
+        self[bytes2str(QUALITY_COLUMN)] = quality
+
+    cpdef get_quality(self) :
+        return self.quality
+
+#    cpdef str reverse_complement(self) :    TODO in C ?
+#        pass
+
+
+cdef class OBI_Nuc_Seq_Stored(OBIView_line) :
+
+    # TODO store the str version of column name macros
+
+    cpdef set_id(self, object id) :
+        self[bytes2str(ID_COLUMN)] = id
+        
+    cpdef object get_id(self) :
+        return self[bytes2str(ID_COLUMN)]
+            
+    cpdef set_definition(self, object definition) :
+        self[bytes2str(DEFINITION_COLUMN)] = definition
+
+    cpdef object get_definition(self) :
+        return self[bytes2str(DEFINITION_COLUMN)]
+
+    cpdef set_sequence(self, object sequence) :
+        self[bytes2str(NUC_SEQUENCE_COLUMN)] = sequence
+
+    cpdef object get_sequence(self) :
+        return self[bytes2str(NUC_SEQUENCE_COLUMN)]
+    
+    cpdef set_quality(self, object quality) :
+        if (type(quality) == list) or (quality is None) :
+            self[bytes2str(QUALITY_COLUMN)] = quality
+        else :  # Quality is in str form
+            (((self.view).columns)[bytes2str(QUALITY_COLUMN)]).set_str_line(self.index, quality)
+
+    cpdef object get_quality(self) :
+        return self[bytes2str(QUALITY_COLUMN)]
+
+    cpdef object get_str_quality(self) :
+        return ((self.view).columns)[bytes2str(QUALITY_COLUMN)].get_str_line(self.index)
+
+#    def __str__(self) :
+#        return self[bytes2str(NUC_SEQUENCE_COLUMN)]    # or not
+    
+#    cpdef str reverse_complement(self) :    TODO in C ?
+#        pass
+
+    # TODO static method to import?
+    

python/obitools3/obidms/_obitaxo.cfiles

@@ -0,0 +1,59 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
+../../../src/obidmscolumn.h
+../../../src/obidmscolumn.c
+../../../src/obidmscolumndir.h
+../../../src/obidmscolumndir.c
+../../../src/obierrno.h
+../../../src/obierrno.c
+../../../src/obilittlebigman.h
+../../../src/obilittlebigman.c
+../../../src/obitypes.h
+../../../src/obitypes.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
+../../../src/utils.h
+../../../src/utils.c

python/obitools3/obidms/_obitaxo.pxd

@@ -0,0 +1,31 @@
+#cython: language_level=3
+
+from .capi.obitaxonomy cimport ecotx_t, OBIDMS_taxonomy_p
+
+from libc.stdint   cimport int32_t
+
+
+cdef class OBI_Taxonomy :
+
+    cdef str name
+    cdef OBIDMS_taxonomy_p pointer
+    
+    cpdef close(self)
+
+ 
+cdef class OBI_Taxon :
+
+    cdef ecotx_t* pointer
+    cdef int32_t  taxid
+    cdef int32_t  rank
+    cdef int32_t  farest
+    cdef ecotx_t* parent
+    cdef str name
+        
+    cpdef int32_t taxid(self)
+    
+    cpdef int32_t rank(self)
+    
+    cpdef int32_t farest(self)
+    
+    cpdef OBI_Taxon parent(self)

python/obitools3/obidms/_obitaxo.pyx

@@ -0,0 +1,65 @@
+#cython: language_level=3
+
+from obitools3.utils   cimport bytes2str, str2bytes
+
+from .capi.obitaxonomy cimport obi_read_taxonomy, \
+                               obi_close_taxonomy, \
+                               obi_taxo_get_taxon_with_taxid
+
+from ._obidms cimport OBIDMS
+
+from cpython.pycapsule cimport PyCapsule_New, PyCapsule_GetPointer
+
+
+cdef class OBI_Taxonomy :
+
+    def __init__(self, OBIDMS dms, str name) :
+        
+        self.name = name
+        self.pointer = obi_read_taxonomy(dms.pointer, str2bytes(name), True)    # TODO discuss
+    
+    
+    def __getitem__(self, object ref):
+        
+        cdef ecotx_t* taxon_p
+        cdef object taxon_capsule
+        
+        if type(ref) == int :
+            taxon_p = obi_taxo_get_taxon_with_taxid(self.pointer, ref)
+            taxon_capsule = PyCapsule_New(taxon_p, NULL, NULL)
+            return OBI_Taxon(taxon_capsule)
+    
+         
+    cpdef close(self) :
+        if (obi_close_taxonomy(self.pointer) < 0) :
+            raise Exception("Error closing the taxonomy")
+        
+
+cdef class OBI_Taxon :    # dict subclass?
+
+    def __init__(self, object taxon_capsule) :
+        
+        cdef ecotx_t* taxon
+        
+        taxon = <ecotx_t*> PyCapsule_GetPointer(taxon_capsule, NULL)   
+        self.pointer = taxon
+        self.taxid = taxon.taxid
+        self.rank = taxon.rank
+        self.farest = taxon.farest
+        self.parent = taxon.parent
+        self.name = bytes2str(taxon.name)
+        
+    cpdef int32_t taxid(self):
+        return self.taxid
+    
+    cpdef int32_t rank(self):
+        return self.rank
+    
+    cpdef int32_t farest(self):
+        return self.farest
+    
+    cpdef OBI_Taxon parent(self):
+        cdef object parent_capsule
+        parent_capsule = PyCapsule_New(self.parent, NULL, NULL)
+        return OBI_Taxon(parent_capsule)
+    

python/obitools3/obidms/capi/obialign.pxd

@@ -0,0 +1,10 @@
+#cython: language_level=3
+
+from ..capi.obiview      cimport Obiview_p
+from ..capi.obidmscolumn cimport OBIDMS_column_p
+
+
+cdef extern from "obi_align.h" nogil:
+
+    int obi_align_one_column(Obiview_p seq_view, OBIDMS_column_p seq_column, Obiview_p score_view, OBIDMS_column_p score_column, double threshold, bint normalize, int reference, bint similarity_mode)
+

python/obitools3/obidms/capi/obidmscolumn.pxd

@@ -11,29 +11,34 @@ from ..capi.obitypes cimport const_char_p, \
                              index_t, \
                              time_t
 
+from libc.stdint cimport uint8_t
+
 
 cdef extern from "obidmscolumn.h" nogil:
     
     struct OBIDMS_column_header_t:
-        bint            little_endian
-        int             header_size
+        size_t          header_size
+        size_t          data_size
         index_t         line_count
         index_t         lines_used
         index_t         nb_elements_per_line
         const_char_p    elements_names
-        OBIType_t       data_type
+        OBIType_t       returned_data_type
+        OBIType_t       stored_data_type
         time_t          creation_date
         obiversion_t    version
         obiversion_t    cloned_from
         const_char_p    name
-        const_char_p    array_name
+        const_char_p    indexer_name
         const_char_p    comments
     
     ctypedef OBIDMS_column_header_t* OBIDMS_column_header_p
 
     struct OBIDMS_column_t:
+        OBIDMS_p               dms
         OBIDMS_column_header_p header
-
+        bint                   writable
+        
     ctypedef OBIDMS_column_t* OBIDMS_column_p
 
     OBIDMS_column_p obi_create_column(OBIDMS_p dms, 
@@ -42,7 +47,8 @@ cdef extern from "obidmscolumn.h" nogil:
                                       index_t nb_lines, 
                                       index_t nb_elements_per_line, 
                                       const_char_p elements_names,
-                                      const_char_p array_name)
+                                      const_char_p indexer_name,
+                                      const_char_p comments)
                 
     OBIDMS_column_p obi_open_column(OBIDMS_p dms, 
                                     const_char_p column_name, 
@@ -51,24 +57,27 @@ cdef extern from "obidmscolumn.h" nogil:
     int obi_close_column(OBIDMS_column_p column)
         
     OBIDMS_column_p obi_clone_column(OBIDMS_p dms, 
+                                     OBIDMS_column_p line_selection,
                                      const_char_p column_name, 
                                      obiversion_t version_number, 
                                      bint clone_data)
     
-    int obi_truncate_and_close_column(OBIDMS_column_p column)
+    int obi_close_column(OBIDMS_column_p column)
     
     obiversion_t obi_column_get_latest_version_from_name(OBIDMS_p dms, 
                                                          const_char_p column_name)
         
     OBIDMS_column_header_p obi_column_get_header_from_name(OBIDMS_p dms, 
-                                                           const_char_p column_name)
+                                                           const_char_p column_name,
+                                                           obiversion_t version_number)
     
-    int obi_unmap_header(OBIDMS_column_header_p header)
-        
-    char* obi_column_format_date(time_t date)
+    int obi_close_header(OBIDMS_column_header_p header)
+
+    int obi_select(OBIDMS_column_p line_selection_column, index_t line_to_grep)
+
 
-    
 cdef extern from "obidmscolumn_int.h" nogil:
+
     int obi_column_set_obiint_with_elt_name(OBIDMS_column_p column, 
                                             index_t line_nb, 
                                             const_char_p element_name, 
@@ -77,7 +86,7 @@ cdef extern from "obidmscolumn_int.h" nogil:
     int obi_column_set_obiint_with_elt_idx(OBIDMS_column_p column, 
                                            index_t line_nb, 
                                            index_t element_idx, 
-                                        obiint_t value)
+                                           obiint_t value)
     
     obiint_t obi_column_get_obiint_with_elt_name(OBIDMS_column_p column, 
                                                  index_t line_nb, 
@@ -87,14 +96,13 @@ cdef extern from "obidmscolumn_int.h" nogil:
                                                 index_t line_nb, 
                                                 index_t element_idx)
 
-
 cdef extern from "obidmscolumn_bool.h" nogil:
 
     int obi_column_set_obibool_with_elt_name(OBIDMS_column_p column, 
                                              index_t line_nb, 
                                              const_char_p element_name, 
                                              obibool_t value)
-    
+
     int obi_column_set_obibool_with_elt_idx(OBIDMS_column_p column, 
                                             index_t line_nb, 
                                             index_t element_idx, 
@@ -114,7 +122,7 @@ cdef extern from "obidmscolumn_char.h" nogil:
                                              index_t line_nb, 
                                              const_char_p element_name, 
                                              obichar_t value)
-    
+
     int obi_column_set_obichar_with_elt_idx(OBIDMS_column_p column, 
                                             index_t line_nb, 
                                             index_t element_idx, 
@@ -134,7 +142,7 @@ cdef extern from "obidmscolumn_float.h" nogil:
                                               index_t line_nb, 
                                               const_char_p element_name, 
                                               obifloat_t value)
-    
+
     int obi_column_set_obifloat_with_elt_idx(OBIDMS_column_p column, 
                                              index_t line_nb, 
                                              index_t element_idx, 
@@ -153,18 +161,81 @@ cdef extern from "obidmscolumn_str.h" nogil:
     int obi_column_set_obistr_with_elt_name(OBIDMS_column_p column, 
                                             index_t line_nb, 
                                             const_char_p element_name, 
-                                            char* value)
-    
+                                            const_char_p value)
+
     int obi_column_set_obistr_with_elt_idx(OBIDMS_column_p column, 
                                            index_t line_nb, 
                                            index_t element_idx, 
-                                           char* value)
+                                           const_char_p value)
     
     const_char_p obi_column_get_obistr_with_elt_name(OBIDMS_column_p column, 
+                                                     index_t line_nb, 
+                                                     const_char_p element_name)
+    
+    const_char_p obi_column_get_obistr_with_elt_idx(OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    index_t element_idx)
+
+cdef extern from "obidmscolumn_seq.h" nogil:
+
+    int obi_column_set_obiseq_with_elt_name(OBIDMS_column_p column, 
+                                            index_t line_nb, 
+                                            const_char_p element_name, 
+                                            const_char_p value)
+
+    int obi_column_set_obiseq_with_elt_idx(OBIDMS_column_p column, 
+                                           index_t line_nb, 
+                                           index_t element_idx, 
+                                           const_char_p value)
+    
+    char* obi_column_get_obiseq_with_elt_name(OBIDMS_column_p column, 
                                               index_t line_nb, 
                                               const_char_p element_name)
     
-    const_char_p obi_column_get_obistr_with_elt_idx(OBIDMS_column_p column, 
+    char* obi_column_get_obiseq_with_elt_idx(OBIDMS_column_p column, 
                                              index_t line_nb, 
                                              index_t element_idx)
 
+
+cdef extern from "obidmscolumn_qual.h" nogil:
+
+    int obi_column_set_obiqual_char_with_elt_name(OBIDMS_column_p column, 
+                                                  index_t line_nb, 
+                                                  const_char_p element_name, 
+                                                  const_char_p value)
+
+    int obi_column_set_obiqual_char_with_elt_idx(OBIDMS_column_p column, 
+                                                 index_t line_nb, 
+                                                 index_t element_idx, 
+                                                 const_char_p value)
+    
+    int obi_column_set_obiqual_int_with_elt_name(OBIDMS_column_p column, 
+                                                  index_t line_nb, 
+                                                  const_char_p element_name, 
+                                                  const uint8_t* value,
+                                                  int value_length)
+    
+    int obi_column_set_obiqual_int_with_elt_idx(OBIDMS_column_p column, 
+                                                index_t line_nb, 
+                                                index_t element_idx, 
+                                                const uint8_t* value,
+                                                int value_length)
+    
+    char* obi_column_get_obiqual_char_with_elt_name(OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    const_char_p element_name)
+    
+    char* obi_column_get_obiqual_char_with_elt_idx(OBIDMS_column_p column, 
+                                                   index_t line_nb, 
+                                                   index_t element_idx)
+
+    const uint8_t* obi_column_get_obiqual_int_with_elt_name(OBIDMS_column_p column, 
+                                                            index_t line_nb, 
+                                                            const_char_p element_name,
+                                                            int* value_length)
+    
+    const uint8_t* obi_column_get_obiqual_int_with_elt_idx(OBIDMS_column_p column, 
+                                                   index_t line_nb, 
+                                                   index_t element_idx,
+                                                   int* value_length)
+

python/obitools3/obidms/capi/obitaxonomy.pxd

@@ -0,0 +1,42 @@
+#cython: language_level=3
+
+from .obitypes     cimport const_char_p
+from .obidms       cimport OBIDMS_p
+from libc.stdint   cimport int32_t
+
+
+cdef extern from "obidms_taxonomy.h" nogil:
+
+    struct OBIDMS_taxonomy_t
+    ctypedef OBIDMS_taxonomy_t* OBIDMS_taxonomy_p
+    
+    struct ecotxnode :
+        int32_t           taxid
+        int32_t           rank
+        int32_t           farest
+        ecotxnode*        parent
+        char*             name
+    
+    ctypedef ecotxnode ecotx_t
+
+
+    OBIDMS_taxonomy_p obi_read_taxonomy(OBIDMS_p dms, const_char_p taxonomy_name, bint read_alternative_names)
+    
+    int obi_close_taxonomy(OBIDMS_taxonomy_p taxonomy)
+    
+    ecotx_t* obi_taxo_get_parent_at_rank(ecotx_t* taxon, int32_t rankidx)
+    
+    ecotx_t* obi_taxo_get_taxon_with_taxid(OBIDMS_taxonomy_p taxonomy, int32_t taxid)
+    
+    bint obi_taxo_is_taxon_under_taxid(ecotx_t* taxon, int32_t other_taxid)
+    
+    ecotx_t* obi_taxo_get_species(ecotx_t* taxon, OBIDMS_taxonomy_p taxonomy)
+    
+    ecotx_t* obi_taxo_get_genus(ecotx_t* taxon, OBIDMS_taxonomy_p taxonomy)
+    
+    ecotx_t* obi_taxo_get_family(ecotx_t* taxon, OBIDMS_taxonomy_p taxonomy)
+    
+    ecotx_t* obi_taxo_get_kingdom(ecotx_t* taxon, OBIDMS_taxonomy_p taxonomy)
+    
+    ecotx_t* obi_taxo_get_superkingdom(ecotx_t* taxon, OBIDMS_taxonomy_p taxonomy)
+    

python/obitools3/obidms/capi/obitypes.pxd

@@ -1,7 +1,8 @@
 #cython: language_level=3
 
 
-from libc.stdint cimport int32_t, int64_t
+from libc.stdint cimport int32_t, int64_t, uint8_t
+
 from posix.types cimport time_t
 
 
@@ -9,20 +10,23 @@ cdef extern from *:
     ctypedef char* const_char_p "const char*"
 
 
-cdef extern from "obidmscolumn.h" nogil:
-
-    ctypedef int32_t obiversion_t
+cdef extern from "encode.h" nogil:
+    bint only_ATGC(const_char_p seq)
 
 
 cdef extern from "obitypes.h" nogil:
 
-    enum OBIType:       # TODO je sais pas si ça sert de declarer le contenu de l'enum
-        OBI_VOID = 0,
+    enum OBIType:
+        OBI_VOID,
         OBI_INT,
         OBI_FLOAT,
         OBI_BOOL,
         OBI_CHAR,
+        OBI_QUAL,
+        OBI_STR,
+        OBI_SEQ,
         OBI_IDX
+
     
     ctypedef OBIType OBIType_t
     
@@ -31,14 +35,21 @@ cdef extern from "obitypes.h" nogil:
     
     ctypedef OBIBool obibool_t
     ctypedef int32_t obiint_t
-    ctypedef double obifloat_t
-    ctypedef char obichar_t  
+    ctypedef double  obifloat_t
+    ctypedef char    obichar_t  
     ctypedef int64_t index_t
 
-    extern obiint_t   OBIInt_NA
-    extern index_t    OBIIdx_NA
-    extern obifloat_t OBIFloat_NA
-    extern obichar_t  OBIChar_NA
-    extern obibool_t  OBIBool_NA
+    ctypedef int32_t obiversion_t
+
+    extern obiint_t      OBIInt_NA
+    extern index_t       OBIIdx_NA
+    extern obifloat_t    OBIFloat_NA
+    extern obichar_t     OBIChar_NA
+    extern obibool_t     OBIBool_NA
+    extern const_char_p  OBISeq_NA
+    extern const_char_p  OBIStr_NA
+    extern const_char_p  OBIQual_char_NA
+    extern uint8_t*      OBIQual_int_NA
+
+    const_char_p name_data_type(int data_type)
 
-    char* name_data_type(int data_type)

python/obitools3/obidms/capi/obiutils.pxd

@@ -0,0 +1,12 @@
+#cython: language_level=3
+
+
+from posix.types cimport time_t
+
+from ..capi.obitypes cimport const_char_p
+
+
+cdef extern from "utils.h" nogil:
+
+    const_char_p obi_format_date(time_t date)
+

python/obitools3/obidms/capi/obiview.pxd

@@ -0,0 +1,280 @@
+#cython: language_level=3
+
+from .obitypes           cimport const_char_p, \
+                                 OBIType_t, \
+                                 obiversion_t, \
+                                 obiint_t, \
+                                 obibool_t, \
+                                 obichar_t, \
+                                 obifloat_t, \
+                                 index_t, \
+                                 time_t
+from ..capi.obidms       cimport OBIDMS_p
+from ..capi.obidmscolumn cimport OBIDMS_column_p
+
+from libc.stdint cimport uint8_t
+
+
+cdef extern from "obiview.h" nogil:
+
+    extern const_char_p VIEW_TYPE_NUC_SEQS
+    extern const_char_p NUC_SEQUENCE_COLUMN
+    extern const_char_p ID_COLUMN
+    extern const_char_p DEFINITION_COLUMN
+    extern const_char_p QUALITY_COLUMN
+
+
+    struct Column_reference_t :
+        const_char_p  column_name
+        obiversion_t  version
+    
+    ctypedef Column_reference_t* Column_reference_p
+
+
+    struct Obiview_infos_t :
+        time_t              creation_date
+        const_char_p        name
+        const_char_p        created_from
+        const_char_p        view_type
+        bint                all_lines
+        Column_reference_t  line_selection
+        index_t             line_count
+        int                 column_count
+        Column_reference_p  column_references
+        const_char_p        comments
+
+    ctypedef Obiview_infos_t* Obiview_infos_p
+
+
+    struct Obiview_t :
+        Obiview_infos_p     infos
+        OBIDMS_p            dms
+        bint                read_only
+        OBIDMS_column_p     line_selection
+        OBIDMS_column_p     new_line_selection
+        OBIDMS_column_p     columns
+
+    ctypedef Obiview_t* Obiview_p
+
+
+    Obiview_p obi_new_view_nuc_seqs(OBIDMS_p dms, const_char_p view_name, Obiview_p view_to_clone, index_t* line_selection, const_char_p comments)
+
+    Obiview_p obi_new_view(OBIDMS_p dms, const_char_p view_name, Obiview_p view_to_clone, index_t* line_selection, const_char_p comments)
+    
+    Obiview_p obi_new_view_cloned_from_name(OBIDMS_p dms, const_char_p view_name, const_char_p view_to_clone_name, index_t* line_selection, const_char_p comments)
+
+    Obiview_p obi_new_view_nuc_seqs_cloned_from_name(OBIDMS_p dms, const_char_p view_name, const_char_p view_to_clone_name, index_t* line_selection, const_char_p comments)
+              
+    Obiview_infos_p obi_view_map_file(OBIDMS_p dms, const char* view_name)
+
+    int obi_view_unmap_file(OBIDMS_p dms, Obiview_infos_p view_infos)
+
+    Obiview_p obi_open_view(OBIDMS_p dms, const_char_p view_name)
+    
+    int obi_view_add_column(Obiview_p view,
+                            const_char_p column_name,
+                            obiversion_t version_number,
+                            OBIType_t data_type,
+                            index_t nb_lines,
+                            index_t nb_elements_per_line,
+                            const_char_p elements_names,
+                            const_char_p indexer_name,
+                            const_char_p comments,
+                            bint create)
+
+    int obi_view_delete_column(Obiview_p view, const_char_p column_name)
+    
+    int obi_select_line(Obiview_p view, index_t line_nb)
+
+    int obi_select_lines(Obiview_p view, index_t* line_nbs)
+        
+    OBIDMS_column_p obi_view_get_column(Obiview_p view, const_char_p column_name)
+
+    OBIDMS_column_p* obi_view_get_pointer_on_column_in_view(Obiview_p view, const_char_p column_name)
+
+    int obi_save_view(Obiview_p view)
+    
+    int obi_close_view(Obiview_p view)
+    
+    int obi_save_and_close_view(Obiview_p view)
+
+    int obi_column_set_obiint_with_elt_name_in_view(Obiview_p view,
+                                                    OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    const_char_p element_name, 
+                                                    obiint_t value)
+
+    int obi_column_set_obiint_with_elt_idx_in_view(Obiview_p view,
+                                                   OBIDMS_column_p column, 
+                                                   index_t line_nb, 
+                                                   index_t element_idx, 
+                                                   obiint_t value)
+    
+    obiint_t obi_column_get_obiint_with_elt_name_in_view(Obiview_p view,
+                                                         OBIDMS_column_p column, 
+                                                         index_t line_nb, 
+                                                         const_char_p element_name)
+    
+    obiint_t obi_column_get_obiint_with_elt_idx_in_view(Obiview_p view,
+                                                        OBIDMS_column_p column, 
+                                                        index_t line_nb, 
+                                                        index_t element_idx)
+
+    int obi_column_set_obibool_with_elt_name_in_view(Obiview_p view,
+                                                     OBIDMS_column_p column, 
+                                                     index_t line_nb, 
+                                                     const_char_p element_name, 
+                                                     obibool_t value)
+
+    int obi_column_set_obibool_with_elt_idx_in_view(Obiview_p view,
+                                                    OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    index_t element_idx, 
+                                                    obibool_t value)
+    
+    obibool_t obi_column_get_obibool_with_elt_name_in_view(Obiview_p view,
+                                                           OBIDMS_column_p column, 
+                                                           index_t line_nb, 
+                                                           const_char_p element_name)
+    
+    obibool_t obi_column_get_obibool_with_elt_idx_in_view(Obiview_p view,
+                                                          OBIDMS_column_p column, 
+                                                          index_t line_nb, 
+                                                          index_t element_idx)
+
+    int obi_column_set_obichar_with_elt_name_in_view(Obiview_p view,
+                                                     OBIDMS_column_p column, 
+                                                     index_t line_nb, 
+                                                     const_char_p element_name, 
+                                                     obichar_t value)
+
+    int obi_column_set_obichar_with_elt_idx_in_view(Obiview_p view,
+                                                    OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    index_t element_idx, 
+                                                    obichar_t value)
+    
+    obichar_t obi_column_get_obichar_with_elt_name_in_view(Obiview_p view,
+                                                           OBIDMS_column_p column, 
+                                                           index_t line_nb, 
+                                                           const_char_p element_name)
+    
+    obichar_t obi_column_get_obichar_with_elt_idx_in_view(Obiview_p view,
+                                                          OBIDMS_column_p column, 
+                                                          index_t line_nb, 
+                                                          index_t element_idx)
+
+    int obi_column_set_obifloat_with_elt_name_in_view(Obiview_p view,
+                                                      OBIDMS_column_p column, 
+                                                      index_t line_nb, 
+                                                      const_char_p element_name, 
+                                                      obifloat_t value)
+
+    int obi_column_set_obifloat_with_elt_idx_in_view(Obiview_p view,
+                                                     OBIDMS_column_p column, 
+                                                     index_t line_nb, 
+                                                     index_t element_idx, 
+                                                     obifloat_t value)
+    
+    obifloat_t obi_column_get_obifloat_with_elt_name_in_view(Obiview_p view,
+                                                             OBIDMS_column_p column, 
+                                                             index_t line_nb, 
+                                                             const_char_p element_name)
+    
+    obifloat_t obi_column_get_obifloat_with_elt_idx_in_view(Obiview_p view,
+                                                            OBIDMS_column_p column, 
+                                                            index_t line_nb, 
+                                                            index_t element_idx)
+
+    int obi_column_set_obiqual_char_with_elt_idx_in_view(Obiview_p view, 
+                                                         OBIDMS_column_p column, 
+                                                         index_t line_nb, 
+                                                         index_t element_idx, 
+                                                         const char* value)
+
+    int obi_column_set_obiqual_int_with_elt_idx_in_view(Obiview_p view, 
+                                                        OBIDMS_column_p column, 
+                                                        index_t line_nb, 
+                                                        index_t element_idx, 
+                                                        const uint8_t* value, 
+                                                        int value_length)
+
+    char* obi_column_get_obiqual_char_with_elt_idx_in_view(Obiview_p view, 
+                                                           OBIDMS_column_p column, 
+                                                           index_t line_nb, 
+                                                           index_t element_idx)
+
+    const uint8_t* obi_column_get_obiqual_int_with_elt_idx_in_view(Obiview_p view, 
+                                                                   OBIDMS_column_p column, 
+                                                                   index_t line_nb, 
+                                                                   index_t element_idx, 
+                                                                   int* value_length)
+
+    int obi_column_set_obiqual_char_with_elt_name_in_view(Obiview_p view, 
+                                                          OBIDMS_column_p column, 
+                                                          index_t line_nb, 
+                                                          const char* element_name, 
+                                                          const char* value)
+
+    int obi_column_set_obiqual_int_with_elt_name_in_view(Obiview_p view, 
+                                                         OBIDMS_column_p column, 
+                                                         index_t line_nb, 
+                                                         const char* element_name, 
+                                                         const uint8_t* value, 
+                                                         int value_length)
+
+    char* obi_column_get_obiqual_char_with_elt_name_in_view(Obiview_p view, 
+                                                            OBIDMS_column_p column, 
+                                                            index_t line_nb, 
+                                                            const char* element_name)
+
+    const uint8_t* obi_column_get_obiqual_int_with_elt_name_in_view(Obiview_p view, 
+                                                                    OBIDMS_column_p column, 
+                                                                    index_t line_nb, 
+                                                                    const char* element_name, 
+                                                                    int* value_length)
+
+    int obi_column_set_obistr_with_elt_name_in_view(Obiview_p view,
+                                                    OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    const_char_p element_name, 
+                                                    const_char_p value)
+
+    int obi_column_set_obistr_with_elt_idx_in_view(Obiview_p view,
+                                                   OBIDMS_column_p column, 
+                                                   index_t line_nb, 
+                                                   index_t element_idx, 
+                                                   const_char_p value)
+    
+    const_char_p obi_column_get_obistr_with_elt_name_in_view(Obiview_p view,
+                                                           OBIDMS_column_p column, 
+                                                           index_t line_nb, 
+                                                           const_char_p element_name)
+    
+    const_char_p obi_column_get_obistr_with_elt_idx_in_view(Obiview_p view,
+                                                            OBIDMS_column_p column, 
+                                                            index_t line_nb, 
+                                                            index_t element_idx)
+
+    int obi_column_set_obiseq_with_elt_name_in_view(Obiview_p view,
+                                                    OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    const_char_p element_name, 
+                                                    const_char_p value)
+
+    int obi_column_set_obiseq_with_elt_idx_in_view(Obiview_p view,
+                                                   OBIDMS_column_p column, 
+                                                   index_t line_nb, 
+                                                   index_t element_idx, 
+                                                   const_char_p value)
+    
+    char* obi_column_get_obiseq_with_elt_name_in_view(Obiview_p view,
+                                                      OBIDMS_column_p column, 
+                                                      index_t line_nb, 
+                                                      const_char_p element_name)
+    
+    char* obi_column_get_obiseq_with_elt_idx_in_view(Obiview_p view,
+                                                     OBIDMS_column_p column, 
+                                                     index_t line_nb, 
+                                                     index_t element_idx)
+

python/obitools3/obigrep.py

@@ -0,0 +1,51 @@
+import sys
+import argparse
+
+from obitools3.obidms._obidms import OBIDMS
+
+if __name__ == '__main__': 
+    
+    parser = argparse.ArgumentParser(description='Pseudo obigrep.')
+    
+    parser.add_argument('-V', '--view', dest='view', type=str,
+                       help='Name of the view that should be considered')    
+
+    parser.add_argument('-N', '--new_view', dest='new_view', type=str,
+                       help='Name of the new view that should be created')    
+    
+#     parser.add_argument('-k', '--key', dest='key', type=str,
+#                        help='Name of the key that should be considered')
+# 
+#     parser.add_argument('-c', '--comp', dest='comparison', type=int,
+#                        help='Comparison to be made: -1:< ; 0:== ; 1:>')
+# 
+#     parser.add_argument('-v', '--value', dest='value', type=object,
+#                        help='Value to be compared')
+
+    args = parser.parse_args()
+    
+    d = OBIDMS('tdms')
+    
+    #condition = 1
+    line_selec = []
+    
+    v = d.open_view(args.view)
+    
+    i = 0
+    for l in v :
+        if l['score'] > 350 : 
+            line_selec.append(i)
+        i+=1
+
+    new_v = d.new_view(args.new_view, view_to_clone=v, line_selection=line_selec, view_type="NUC_SEQS_VIEW", comments="obigrep "+args.view+" to "+args.new_view) #args.key+" "+str(args.comparison)+" "+str(args.value)+" "+)
+
+    print("\n")
+    print(new_v.__repr__())
+
+    v.save_and_close()
+    new_v.save_and_close()
+    
+    d.close()
+
+    print("\nDone.")
+    

python/obitools3/obihead.py

@@ -0,0 +1,43 @@
+import sys
+import argparse
+
+from obitools3.obidms._obidms import OBIDMS
+
+if __name__ == '__main__': 
+    
+    parser = argparse.ArgumentParser(description='Pseudo obihead.')
+    
+    parser.add_argument('-V', '--view', dest='view', type=str,
+                       help='Name of the view that should be considered')    
+
+    parser.add_argument('-N', '--new_view', dest='new_view', type=str,
+                       help='Name of the new view that should be created')    
+    
+    parser.add_argument('-n', '--nb', dest='nb_lines', type=int,
+                       help='Number of lines that should be taken')    
+    
+
+    args = parser.parse_args()
+    
+    d = OBIDMS('tdms')
+    
+    #condition = 1
+    line_selec = []
+    
+    v = d.open_view(args.view)
+    
+    for i in range(0, args.nb_lines) :
+        line_selec.append(i)
+        
+    new_v = d.new_view(args.new_view, view_to_clone=v, line_selection=line_selec, view_type="NUC_SEQS_VIEW", comments="obihead "+str(args.nb_lines)+", "+args.view+" to "+args.new_view) #args.key+" "+str(args.comparison)+" "+str(args.value)+" "+)
+
+    print("\n")
+    print(new_v.__repr__())
+
+    v.save_and_close()
+    new_v.save_and_close()
+    
+    d.close()
+
+    print("\nDone.")
+    

python/obitools3/obiimport.py

@@ -0,0 +1,199 @@
+import sys
+import argparse
+import time
+
+from obitools3.obidms._obidms import OBIDMS
+
+
+def bufferedRead(fileobj,size=209715200):   ## 200 MB
+   buffer = fileobj.readlines(size)
+   while buffer:
+       for l in buffer:
+           yield l
+       buffer = fileobj.readlines(size)
+
+
+if __name__ == '__main__': 
+    
+    parser = argparse.ArgumentParser(description='Convert a fasta file in an OBIDMS.')
+    
+    parser.add_argument('-i', '--input', dest='input_file', type=str,
+                       help='Name of the file containing the sequences')
+
+    args = parser.parse_args()
+
+    d = OBIDMS('tdms')
+    
+    view = d.new_view('uniq view', view_type="NUC_SEQS_VIEW")
+
+#    for i in range(35000000) :
+#        if (not (i%500000)) :
+#            print(str(time.time())+'\t'+str(i))
+#        id = "@HWI-D00405:142:C71BAANXX:4:1101:1234:2234_CONS_SUB_SUB_"+str(i)
+#        view[i].set_id(id)
+#        if id != view[i]["ID"] :
+#            print("nope", id, view[i]["ID"])
+    
+    input_file = open(args.input_file, 'r')
+    input_file_buffered = bufferedRead(input_file)
+    
+#         
+#     if args.input_file[-1:] == "a" :
+#     
+#         i = 0
+#         next = False
+#         first = True
+#         
+#         for line in input_file :
+#             
+#             if line[0] == ">" :
+#                 
+#                 if not first :
+#                     # save seq
+#                     #print(i, id, seq)
+#                     view[i].set_sequence(seq)
+#                     i+=1
+#                 
+#                 first = False
+#                     
+#                 #id = line.split(" ", 1)[0][1:]
+#                 #rest = (line[:-1].split(" ", 1)[1]).split(";")
+#                 #view[i].set_id(id)
+#                 
+# #                 description = ""
+# #                 for j in range(len(rest)) :
+# #                     if "=" in rest[j] :
+# #                         rest[j] = rest[j].strip()
+# #                         rest[j] = rest[j].split("=", 1)
+# #                         column_name = rest[j][0]
+# #                         v = rest[j][1]
+# #                         if ((not v.isalpha()) and (v.isalnum())) :
+# #                             conv_v = int(v)
+# #                         elif (v == "True") or (v == "False") :
+# #                             conv_v = bool(v)
+# #                         else :
+# #                             f = True
+# #                             for letter in v :
+# #                                 if ((not letter.isalnum()) or (letter != ".")) :
+# #                                     f = False
+# #                             if f :
+# #                                 conv_v = float(v)
+# #                             else : 
+# #                                 conv_v = v
+# #                         view[i][column_name] = conv_v
+# #                     else :
+# #                         description+=rest[j]
+# #                 
+# #                 if description != "" :
+# #                     description = description.strip()
+# #                     view[i].set_description(description)
+#                     
+#                 #print(id)
+#                 #print(rest)
+#                 #print(description)
+#                 
+#                 next = True
+#                 
+#             elif next == True :
+#                 
+#      #           if not (i % 1E5) :
+#      #               print(i)
+#                 
+#                 seq = line[:-1]
+#                 next = False
+#             
+#             elif not next :
+#                 
+#                 seq += line[:-1]
+# 
+# 
+#     elif args.input_file[-1:] == "q" :
+#         
+#         i = 0
+#         l = 0
+#         next = False
+#
+    l=0
+    i=0
+#     while (True):
+#         l+=1
+#         line = input_file.readline()
+#         if line=="":
+#             break
+    for line in input_file_buffered :
+# 
+             #if i > 1E7 :
+#             #    print('hmm?')
+#                         
+#        if i == 6000000 :
+#            break
+#             
+        if l%4 == 0 :
+#                
+            if (not (i%500000)) :
+                print(str(time.time())+'\t'+str(i))
+# # 
+# #                 #print("header", line)
+# # 
+            id = line.split(" ", 1)[0][1:]
+#            print(id)
+# #                 #rest = (line[:-1].split(" ", 1)[1]).split(";")
+            view[i].set_id(id)
+#            print(view[i]["ID"])
+# 
+#            i+=1
+        
+#        l+=1
+#                 
+# #                 description = ""
+# #                 for j in range(len(rest)) :
+# #                     if "=" in rest[j] :
+# #                         rest[j] = rest[j].strip()
+# #                         rest[j] = rest[j].split("=", 1)
+# #                         column_name = rest[j][0]
+# #                         #print("COLUMN", column_name)
+# #                         v = rest[j][1]
+# #                         if (v == "") and (column_name in view) and (view[column_name].get_data_type() == "OBI_SEQ") :
+# #                             #print(">>>>>>YUP")
+# #                             conv_v = "aa"
+# #                         else :
+# #                             if ((not v.isalpha()) and (v.isalnum())) :
+# #                                 conv_v = int(v)
+# #                             elif (v == "True") or (v == "False") :
+# #                                 conv_v = bool(v)
+# #                             else :
+# #                                 f = True
+# #                                 for letter in v :
+# #                                     if ((not letter.isalnum()) or (letter != ".")) :
+# #                                         f = False
+# #                                 if f :
+# #                                     conv_v = float(v)
+# #                                 else :
+# #                                     conv_v = v
+# #                         view[i][column_name] = conv_v
+# #                     else :
+# #                         description+=rest[j]
+# #                 
+# #                 if description != "" :
+# #                     description = description.strip()
+# #                     view[i].set_description(description)
+#                 
+        elif l%4 == 1 :
+# #                
+            seq = line[:-1]
+#                 #print("seq", seq)
+            view[i].set_sequence(seq)
+            i+=1
+#                         
+        l+=1
+#     
+#     
+    input_file.close()
+
+    #print(view)
+    print(view.__repr__())
+
+    view.save_and_close()
+    d.close()
+
+    print("Done.")

python/obitools3/parsers/fasta.pxd

@@ -0,0 +1,9 @@
+#cython: language_level=3
+
+from ..utils cimport str2bytes
+from .header cimport parseHeader
+from ..files.universalopener cimport uopen
+from ..files.linebuffer cimport LineBuffer
+
+
+    

python/obitools3/parsers/fasta.pyx

@@ -0,0 +1,55 @@
+#cython: language_level=3
+
+'''
+Created on 30 mars 2016
+
+@author: coissac
+'''
+
+
+def fastaIterator(lineiterator, int buffersize=100000000):
+    cdef LineBuffer lb
+    cdef str        ident
+    cdef str        definition
+    cdef dict       tags
+    cdef list       s
+    cdef bytes      sequence
+    cdef bytes      quality
+    
+    if isinstance(lineiterator,(str,bytes)):
+        lineiterator=uopen(lineiterator)
+    
+    if isinstance(lineiterator, LineBuffer):
+        lb=lineiterator
+    else:
+        lb=LineBuffer(lineiterator,buffersize)
+        
+    i = iter(lb)
+    line = next(i)
+    
+    while True:
+        ident,tags,definition = parseHeader(line)
+        s = []
+        line = next(i)
+        
+        try:
+            while line[0]!='>':
+                s.append(str2bytes(line)[0:-1])
+                line = next(i)
+        except StopIteration:
+            pass
+        
+        sequence  = b"".join(s)
+        quality   = None
+
+        yield { "id"         : ident,
+                "definition" : definition,
+                "sequence"   : sequence,
+                "quality"    : quality,
+                "tags"       : tags,
+                "annotation" : {}
+              }
+        
+    
+        
+        

python/obitools3/parsers/fastq.pxd

@@ -0,0 +1,10 @@
+#cython: language_level=3
+
+from ..utils cimport str2bytes
+
+from .header cimport parseHeader
+from ..files.universalopener cimport uopen
+from ..files.linebuffer cimport LineBuffer
+
+
+    

python/obitools3/parsers/fastq.pyx

@@ -0,0 +1,40 @@
+#cython: language_level=3
+
+'''
+Created on 30 mars 2016
+
+@author: coissac
+'''
+
+def fastqIterator(lineiterator, int buffersize=100000000):
+    cdef LineBuffer lb
+    cdef str        ident
+    cdef str        definition
+    cdef dict       tags
+    cdef bytes      sequence
+    cdef bytes      quality
+    
+    if isinstance(lineiterator,(str,bytes)):
+        lineiterator=uopen(lineiterator)
+    
+    if isinstance(lineiterator, LineBuffer):
+        lb=lineiterator
+    else:
+        lb=LineBuffer(lineiterator,buffersize)
+        
+    i = iter(lb)
+    for line in i:
+        ident,tags,definition = parseHeader(line)
+        sequence  = str2bytes(next(i)[0:-1])
+        next(i)
+        quality   = str2bytes(next(i)[0:-1])
+
+        yield { "id"         : ident,
+                "definition" : definition,
+                "sequence"   : sequence,
+                "quality"    : quality,
+                "tags"       : tags,
+                "annotation" : {}
+              }
+        
+        

python/obitools3/parsers/header.pxd

@@ -0,0 +1,5 @@
+#cython: language_level=3
+
+cdef object __etag__(str x)
+
+cpdef tuple parseHeader(str header)

python/obitools3/parsers/header.pyx

@@ -0,0 +1,85 @@
+#cython: language_level=3
+
+'''
+Created on 25 mars 2016
+
+@author: coissac
+'''
+
+import re
+
+__ret__         = re.compile('''(([^ ]+)=('[^']*'|"[^"]*"|[^;]+); *)+?''')
+__re_int__      = re.compile("^[+-]?[0-9]+$")
+__re_float__    = re.compile("^[+-]?[0-9]+(\.[0-9]*)?([eE][+-]?[0-9]+)?$")
+__re_str__      = re.compile("""^"[^"]*"|'[^']*'$""")
+__re_dict__     = re.compile("""^\{\ *
+                                   (
+                                       ("[^"]*"|'[^']*')
+                                        \ *:\ *
+                                       ([^,}]+|
+                                        "[^"]*"|
+                                        '[^']*'
+                                       )
+                                   )?
+                                   (\ *,\ *
+                                       ("[^"]*"|'[^']*')
+                                        \ *:\ *
+                                       ([^,}]+|
+                                        "[^"]*"|
+                                        '[^']*'
+                                       )
+                                    )*\ *\}$""", re.VERBOSE)
+
+__re_val__ = re.compile("""(("[^"]*"|'[^']*') *: *([^,}]+|"[^"]*"|'[^']*') *[,}] *)""")
+
+cdef object __etag__(str x):
+    cdef list elements
+    cdef tuple i
+    
+    if __re_int__.match(x):
+        v=int(x)
+    elif __re_float__.match(x):
+        v=float(x)
+    elif __re_str__.match(x):
+        v=x[1:-1]
+    elif x=='None':
+        v=None
+    elif x=='False':
+        v=False
+    elif x=='True':
+        v=True
+    elif __re_dict__.match(x):
+        elements=__re_val__.findall(x)
+        v=dict([(i[1][1:-1],__etag__(i[2])) for i in elements])
+    else:
+        v=x
+    return v
+
+cpdef tuple parseHeader(str header):
+    cdef list m 
+    cdef dict tags 
+    cdef str definition
+    cdef str ident
+    cdef str second
+    
+    m=header[1:-1].split(maxsplit=1)
+   
+    ident=m[0]
+    
+    if len(m)==1:
+        tags={}
+        definition='' 
+    else:
+        second=m[1]
+        m = __ret__.findall(second)
+        
+        if m:
+            tags = dict([(a[1],__etag__(a[2])) for a in m])
+            definition = second.split(m[-1][0],1)[1].strip()
+        else:
+            tags = {}
+            definition = second.strip() 
+        
+    return ident,tags,definition
+    
+

python/obitools3/unit_tests.py

@@ -10,9 +10,8 @@ from obitools3.obidms._obidms import OBIDMS
 
 LINE_COUNT_FOR_TEST_COLUMN = 10000         # TODO randomize?
 SMALLER_LINE_COUNT_FOR_TEST_COLUMN = 1000  # TODO randomize?
-NB_ELEMENTS_PER_LINE = 20                  # TODO randomize?
+NB_ELEMENTS_PER_LINE = 10                  # TODO randomize?
 DMS_NAME = "unit_test_dms"
-DATA_TYPES = ['OBI_INT', 'OBI_FLOAT', 'OBI_BOOL', 'OBI_CHAR', 'OBI_IDX']
 
 
 def create_test_obidms():
@@ -22,26 +21,24 @@ def create_test_obidms():
     return (dms, dms_name, dms_dir_name)
 
 
-def create_test_column(dms, data_type_code, multiple_elements_per_line=False):
-    data_types = DATA_TYPES
-    data_type_code = data_type_code
-    data_type_str = data_types[data_type_code-1]
-    col_name = "unit_test_"+data_type_str
-        
+def create_test_column(dms, data_type, multiple_elements_per_line=False):
+    
+    col_name = "unit_test_"+data_type
+
     if multiple_elements_per_line :
         elts_names = elements_names()
         col = dms.open_column(col_name, 
                               create=True, 
-                              data_type=data_type_code, 
+                              type=data_type, 
                               nb_elements_per_line=NB_ELEMENTS_PER_LINE,
                               elements_names=elts_names)
-        return (col, col_name, elts_names, data_type_str)
+        return (col, col_name, elts_names)
 
     else :
         col = dms.open_column(col_name, 
                               create=True, 
-                              data_type=data_type_code)
-        return (col, col_name, data_type_str)
+                              type=data_type)
+        return (col, col_name)
 
 
 def elements_names():
@@ -58,12 +55,15 @@ def random_obivalue(data_type):
     elif data_type == "OBI_BOOL" :
         return randint(0,1)
     elif data_type == "OBI_CHAR" :
-        nucs = 'atgc'
-        return bytes(nucs[randint(0,3)], 'utf-8')
-    elif data_type == "OBI_IDX" :
-        length = randint(1,500)
+        return choice(string.ascii_lowercase)
+    elif data_type == "OBI_STR" :
+        length = randint(1,200)
         randoms = ''.join(choice(string.ascii_lowercase) for i in range(length))
         return randoms
+    elif data_type == "OBI_SEQ" :
+        length = randint(1,200)
+        randoms = ''.join(choice("atgcryswkmdbhvn") for i in range(length))
+        return randoms
 
 class OBIDMS_Column_TestCase(unittest.TestCase):
     def tearDown(self):
@@ -71,10 +71,10 @@ class OBIDMS_Column_TestCase(unittest.TestCase):
         self.dms.close()
         shutil.rmtree(self.dms_dir_name, ignore_errors=True)
     def test_OBIDMS_column_type(self):
-        assert self.col.get_data_type() == self.data_type_str, 'Wrong data type associated with column'
+        assert self.col.get_data_type() == self.data_type, 'Wrong data type associated with column'
     def test_OBIDMS_column_cloning(self):
         for i in range(LINE_COUNT_FOR_TEST_COLUMN):
-            self.col[i]= random_obivalue(self.data_type_str)
+            self.col[i]= random_obivalue(self.data_type)
         self.col.close()
         clone = self.dms.open_column(self.col_name, clone=True)
         self.col = self.dms.open_column(self.col_name)
@@ -86,10 +86,21 @@ class OBIDMS_Column_TestCase(unittest.TestCase):
         clone.close()
     def test_OBIDMS_column_set_and_get(self):
         for i in range(LINE_COUNT_FOR_TEST_COLUMN):
-            v = random_obivalue(self.data_type_str)
+            v = random_obivalue(self.data_type)
             self.col[i] = v
             assert self.col[i] == v, "Different value than the set value"
             assert self.col[i] is not None, "None value"
+    def test_OBIDMS_referring_column(self):
+        for i in range(LINE_COUNT_FOR_TEST_COLUMN):
+            self.col[i] = random_obivalue(self.data_type)
+        ref_col = self.dms.open_column(self.col_name, referring=True)
+        j = 0
+        for i in range(LINE_COUNT_FOR_TEST_COLUMN):
+            if i%2 :    # TODO randomize
+                ref_col.grep_line(i)
+                assert ref_col[j] == self.col[i], "Different value in original column and returned by referring column"
+                assert ref_col[j] is not None, "None value"
+                j+=1
 
 
 class OBIDMS_Column_multiple_elements_TestCase(OBIDMS_Column_TestCase):
@@ -98,7 +109,7 @@ class OBIDMS_Column_multiple_elements_TestCase(OBIDMS_Column_TestCase):
         for i in range(SMALLER_LINE_COUNT_FOR_TEST_COLUMN):
             v = {}
             for e in self.elts_names :
-                v[e] = random_obivalue(self.data_type_str)
+                v[e] = random_obivalue(self.data_type)
             self.col[i] = v
         self.col.close()
         clone = self.dms.open_column(self.col_name, clone=True)
@@ -112,7 +123,7 @@ class OBIDMS_Column_multiple_elements_TestCase(OBIDMS_Column_TestCase):
     def test_OBIDMS_column_set_and_get_with_elements_names(self):
         for i in range(SMALLER_LINE_COUNT_FOR_TEST_COLUMN):
             for e in range(NB_ELEMENTS_PER_LINE) :
-                v = random_obivalue(self.data_type_str)
+                v = random_obivalue(self.data_type)
                 self.col.set_item(i, self.elts_names[e], v)
                 assert self.col.get_item(i, self.elts_names[e]) == v, "Different value than the set value"
                 assert self.col.get_item(i, self.elts_names[e]) is not None, "None value"
@@ -120,141 +131,168 @@ class OBIDMS_Column_multiple_elements_TestCase(OBIDMS_Column_TestCase):
         for i in range(SMALLER_LINE_COUNT_FOR_TEST_COLUMN):
             v = {}
             for e in self.elts_names :
-                v[e] = random_obivalue(self.data_type_str)
+                v[e] = random_obivalue(self.data_type)
             self.col[i] = v
             assert self.col[i] == v, "Different value than the set value"
             assert self.col[i] is not None, "None value"
+    def test_OBIDMS_referring_column(self):
+        for i in range(SMALLER_LINE_COUNT_FOR_TEST_COLUMN):
+            v = {}
+            for e in self.elts_names :
+                v[e] = random_obivalue(self.data_type)
+            self.col[i] = v
+        ref_col = self.dms.open_column(self.col_name, referring=True)
+        j = 0
+        for i in range(SMALLER_LINE_COUNT_FOR_TEST_COLUMN):
+            if i%2 :    # TODO randomize
+                ref_col.grep_line(i)
+                assert ref_col[j] == self.col[i], "Different value in original column and returned by referring column"
+                assert ref_col[j] is not None, "None value"
+                j+=1
+        ref_col.close()
 
 
 class OBIDMS_Column_OBI_INT_TestCase(OBIDMS_Column_TestCase):
     def setUp(self):
-        self.data_type_code = 1
+        self.data_type = 'OBI_INT'
         self.dms, \
         self.dms_name, \
         self.dms_dir_name = create_test_obidms()
         self.col, \
-        self.col_name, \
-        self.data_type_str = create_test_column(self.dms, 
-                                                self.data_type_code)
+        self.col_name = create_test_column(self.dms, 
+                                           self.data_type)
        
 
 class OBIDMS_Column_OBI_INT_multiple_elements_TestCase(OBIDMS_Column_multiple_elements_TestCase):
     def setUp(self):
-        self.data_type_code = 1
+        self.data_type = 'OBI_INT'
         self.dms, \
         self.dms_name, \
         self.dms_dir_name = create_test_obidms()
         self.col, \
         self.col_name, \
-        self.elts_names, \
-        self.data_type_str = create_test_column(self.dms, 
-                                                self.data_type_code,
-                                                multiple_elements_per_line=True)
+        self.elts_names = create_test_column(self.dms, 
+                                             self.data_type,
+                                             multiple_elements_per_line=True)
 
 
 class OBIDMS_Column_OBI_FLOAT_TestCase(OBIDMS_Column_TestCase):
     def setUp(self):
-        self.data_type_code = 2
+        self.data_type = 'OBI_FLOAT'
         self.dms, \
         self.dms_name, \
         self.dms_dir_name = create_test_obidms()
         self.col, \
-        self.col_name, \
-        self.data_type_str = create_test_column(self.dms, 
-                                                self.data_type_code)
+        self.col_name = create_test_column(self.dms, 
+                                           self.data_type)
        
 
 class OBIDMS_Column_OBI_FLOAT_multiple_elements_TestCase(OBIDMS_Column_multiple_elements_TestCase):
     def setUp(self):
-        self.data_type_code = 2
+        self.data_type = 'OBI_FLOAT'
         self.dms, \
         self.dms_name, \
         self.dms_dir_name = create_test_obidms()
         self.col, \
         self.col_name, \
-        self.elts_names, \
-        self.data_type_str = create_test_column(self.dms, 
-                                                self.data_type_code,
-                                                multiple_elements_per_line=True)
+        self.elts_names = create_test_column(self.dms, 
+                                             self.data_type,
+                                             multiple_elements_per_line=True)
 
 
 class OBIDMS_Column_OBI_BOOL_TestCase(OBIDMS_Column_TestCase):
     def setUp(self):
-        self.data_type_code = 3
+        self.data_type = 'OBI_BOOL'
         self.dms, \
         self.dms_name, \
         self.dms_dir_name = create_test_obidms()
         self.col, \
-        self.col_name, \
-        self.data_type_str = create_test_column(self.dms, 
-                                                self.data_type_code)
+        self.col_name = create_test_column(self.dms, 
+                                           self.data_type)
        
 
 class OBIDMS_Column_OBI_BOOL_multiple_elements_TestCase(OBIDMS_Column_multiple_elements_TestCase):
     def setUp(self):
-        self.data_type_code = 3
+        self.data_type = 'OBI_BOOL'
         self.dms, \
         self.dms_name, \
         self.dms_dir_name = create_test_obidms()
         self.col, \
         self.col_name, \
-        self.elts_names, \
-        self.data_type_str = create_test_column(self.dms, 
-                                                self.data_type_code,
-                                                multiple_elements_per_line=True)
+        self.elts_names = create_test_column(self.dms, 
+                                             self.data_type,
+                                             multiple_elements_per_line=True)
 
 
 class OBIDMS_Column_OBI_CHAR_TestCase(OBIDMS_Column_TestCase):
     def setUp(self):
-        self.data_type_code = 4
+        self.data_type = 'OBI_CHAR'
         self.dms, \
         self.dms_name, \
         self.dms_dir_name = create_test_obidms()
         self.col, \
-        self.col_name, \
-        self.data_type_str = create_test_column(self.dms, 
-                                                self.data_type_code)
+        self.col_name = create_test_column(self.dms, 
+                                                self.data_type)
        
 
 class OBIDMS_Column_OBI_CHAR_multiple_elements_TestCase(OBIDMS_Column_multiple_elements_TestCase):
     def setUp(self):
-        self.data_type_code = 4
+        self.data_type = 'OBI_CHAR'
         self.dms, \
         self.dms_name, \
         self.dms_dir_name = create_test_obidms()
         self.col, \
         self.col_name, \
-        self.elts_names, \
-        self.data_type_str = create_test_column(self.dms, 
-                                                self.data_type_code,
-                                                multiple_elements_per_line=True)
+        self.elts_names = create_test_column(self.dms, 
+                                             self.data_type,
+                                             multiple_elements_per_line=True)
 
 
 class OBIDMS_Column_OBI_STR_TestCase(OBIDMS_Column_TestCase):
     def setUp(self):
-        self.data_type_code = 5
+        self.data_type = 'OBI_STR'
         self.dms, \
         self.dms_name, \
         self.dms_dir_name = create_test_obidms()
         self.col, \
-        self.col_name, \
-        self.data_type_str = create_test_column(self.dms, 
-                                                self.data_type_code)
+        self.col_name = create_test_column(self.dms, 
+                                           self.data_type)
        
 
 class OBIDMS_Column_OBI_STR_multiple_elements_TestCase(OBIDMS_Column_multiple_elements_TestCase):
     def setUp(self):
-        self.data_type_code = 5
+        self.data_type = 'OBI_STR'
         self.dms, \
         self.dms_name, \
         self.dms_dir_name = create_test_obidms()
         self.col, \
         self.col_name, \
-        self.elts_names, \
-        self.data_type_str = create_test_column(self.dms, 
-                                                self.data_type_code,
-                                                multiple_elements_per_line=True)
+        self.elts_names = create_test_column(self.dms, 
+                                             self.data_type,
+                                             multiple_elements_per_line=True)
 
+class OBIDMS_Column_OBI_SEQ_TestCase(OBIDMS_Column_TestCase):
+    def setUp(self):
+        self.data_type = 'OBI_SEQ'
+        self.dms, \
+        self.dms_name, \
+        self.dms_dir_name = create_test_obidms()
+        self.col, \
+        self.col_name = create_test_column(self.dms, 
+                                           self.data_type)
+       
+
+class OBIDMS_Column_OBI_SEQ_multiple_elements_TestCase(OBIDMS_Column_multiple_elements_TestCase):
+    def setUp(self):
+        self.data_type = 'OBI_SEQ'
+        self.dms, \
+        self.dms_name, \
+        self.dms_dir_name = create_test_obidms()
+        self.col, \
+        self.col_name, \
+        self.elts_names = create_test_column(self.dms, 
+                                             self.data_type,
+                                             multiple_elements_per_line=True)
 
 if __name__ == '__main__': 
      unittest.main(verbosity=2, defaultTest=["OBIDMS_Column_OBI_INT_TestCase",
@@ -266,6 +304,8 @@ if __name__ == '__main__':
                                              "OBIDMS_Column_OBI_CHAR_TestCase",
                                              "OBIDMS_Column_OBI_CHAR_multiple_elements_TestCase",
                                              "OBIDMS_Column_OBI_STR_TestCase",
-                                             "OBIDMS_Column_OBI_STR_multiple_elements_TestCase"])
+                                             "OBIDMS_Column_OBI_STR_multiple_elements_TestCase",
+                                             "OBIDMS_Column_OBI_SEQ_TestCase",
+                                             "OBIDMS_Column_OBI_SEQ_multiple_elements_TestCase"])
 
 

python/obitools3/utils.pxd

@@ -1,5 +1,6 @@
 #cython: language_level=3
 
+
 cdef bytes str2bytes(str string)
 
 cdef str bytes2str(bytes string)

python/obitools3/utils.pyx

@@ -1,7 +1,26 @@
 #cython: language_level=3
 
+
 cdef bytes str2bytes(str string):
+    """
+    Short cut to convert ascii encoded python string (str) to bytes 
+    which can be easily converted to C-strings. 
+    
+        @param string: the python string to be converted.
+        @type string: str
+        @return a transcoded string
+        @rtype: bytes 
+    """
     return string.encode('ascii')
 
 cdef str bytes2str(bytes string):
-    return string.decode('ascii')
+    """
+    Short cut to convert bytes (C-strings) to ascii encoded python string (str).
+    
+        @param string: the binary (C-string) string to be converted.
+        @type string: bytes
+        @return an ascii transcoded string
+        @rtype: str 
+    """
+    return string.decode('ascii')
+

python/obitools3/version.py

@@ -1,5 +1,5 @@
-major = 1
-minor = 1
-serial= '16'
+major = 0
+minor = 0
+serial= '0'
 
 version ="%d.%02d.%s" % (major,minor,serial)

requirements.txt

@@ -1,5 +1,5 @@
 --extra-index-url https://pypi.python.org/simple/
-Cython>=0.21
+Cython==0.23.5
 Sphinx>=1.2.0
 ipython>=3.0.0
 breathe>=4.0.0

src/_sse.h

@@ -0,0 +1,961 @@
+#ifndef _SSE_H_
+#define _SSE_H_
+
+#include <string.h>
+
+#include <inttypes.h>
+#ifdef __SSE2__
+#include <xmmintrin.h>
+#else
+typedef long long __m128i __attribute__ ((__vector_size__ (16), __may_alias__));
+#endif /* __SSE2__ */
+
+#ifndef MAX
+#define MAX(x,y) (((x)>(y)) ? (x):(y))
+#define MIN(x,y) (((x)<(y)) ? (x):(y))
+#endif
+
+#define ALIGN __attribute__((aligned(16)))
+typedef __m128i vUInt8;
+typedef __m128i vInt8;
+
+typedef __m128i vUInt16;
+typedef __m128i vInt16;
+
+typedef __m128i vUInt64;
+
+typedef union
+{
+     __m128i i;
+     int64_t  s64[ 2];
+     int16_t  s16[ 8];
+     int8_t   s8 [16];
+     uint8_t  u8 [16];
+     uint16_t u16[8 ];
+	 uint32_t u32[4 ];
+     uint64_t u64[2 ];
+} um128;
+
+typedef union
+    {
+	    vUInt8 m;
+	    uint8_t c[16];
+    } uchar_v;
+
+typedef union
+	{
+		vUInt16 m;
+		uint16_t c[8];
+	} ushort_v;
+
+typedef union
+	{
+		vUInt64 m;
+		uint64_t c[2];
+	} uint64_v;
+
+
+#ifdef __SSE2__
+
+static inline int8_t _s2_extract_epi8(__m128i r, const int p)
+{
+#define ACTIONP(r,x) return _mm_extract_epi16(r,x) & 0xFF
+#define ACTIONI(r,x) return _mm_extract_epi16(r,x) >> 8
+	switch (p) {
+	case  0:  ACTIONP(r,0);
+	case  1:  ACTIONI(r,0);
+	case  2:  ACTIONP(r,1);
+	case  3:  ACTIONI(r,1);
+	case  4:  ACTIONP(r,2);
+	case  5:  ACTIONI(r,2);
+	case  6:  ACTIONP(r,3);
+	case  7:  ACTIONI(r,3);
+	case  8:  ACTIONP(r,4);
+	case  9:  ACTIONI(r,4);
+	case 10:  ACTIONP(r,5);
+	case 11:  ACTIONI(r,5);
+	case 12:  ACTIONP(r,6);
+	case 13:  ACTIONI(r,6);
+	case 14:  ACTIONP(r,7);
+	case 15:  ACTIONI(r,7);
+	}
+#undef ACTIONP
+#undef ACTIONI
+
+	return 0;
+}
+
+static inline __m128i _s2_max_epi8(__m128i a, __m128i b)
+{
+	__m128i mask  = _mm_cmpgt_epi8( a, b );
+	a = _mm_and_si128   (a,mask );
+	b = _mm_andnot_si128(mask,b);
+	return _mm_or_si128(a,b);
+}
+
+static inline __m128i _s2_min_epi8(__m128i a, __m128i b)
+{
+	__m128i mask  = _mm_cmplt_epi8( a, b );
+	a = _mm_and_si128   (a,mask );
+	b = _mm_andnot_si128(mask,b);
+	return _mm_or_si128(a,b);
+}
+
+static inline __m128i _s2_insert_epi8(__m128i r, int b, const int p)
+{
+#define ACTIONP(r,x) return _mm_insert_epi16(r,(_mm_extract_epi16(r,x) & 0xFF00) | (b & 0x00FF),x)
+#define ACTIONI(r,x) return _mm_insert_epi16(r,(_mm_extract_epi16(r,x) & 0x00FF) | ((b << 8)& 0xFF00),x)
+	switch (p) {
+	case  0:  ACTIONP(r,0);
+	case  1:  ACTIONI(r,0);
+	case  2:  ACTIONP(r,1);
+	case  3:  ACTIONI(r,1);
+	case  4:  ACTIONP(r,2);
+	case  5:  ACTIONI(r,2);
+	case  6:  ACTIONP(r,3);
+	case  7:  ACTIONI(r,3);
+	case  8:  ACTIONP(r,4);
+	case  9:  ACTIONI(r,4);
+	case 10:  ACTIONP(r,5);
+	case 11:  ACTIONI(r,5);
+	case 12:  ACTIONP(r,6);
+	case 13:  ACTIONI(r,6);
+	case 14:  ACTIONP(r,7);
+	case 15:  ACTIONI(r,7);
+	}
+#undef ACTIONP
+#undef ACTIONI
+
+	return _mm_setzero_si128();
+}
+
+// Fill a SSE Register with 16 time the same 8bits integer value
+#define _MM_SET1_EPI8(x)        _mm_set1_epi8(x)
+#define _MM_INSERT_EPI8(r,x,i)	_s2_insert_epi8((r),(x),(i))
+#define _MM_CMPEQ_EPI8(x,y)     _mm_cmpeq_epi8((x),(y))
+#define _MM_CMPGT_EPI8(x,y)     _mm_cmpgt_epi8((x),(y))
+#define _MM_CMPLT_EPI8(x,y)     _mm_cmplt_epi8((x),(y))
+#define _MM_MAX_EPI8(x,y)       _s2_max_epi8((x),(y))
+#define _MM_MIN_EPI8(x,y)       _s2_min_epi8((x),(y))
+#define _MM_ADD_EPI8(x,y)       _mm_add_epi8((x),(y))
+#define _MM_SUB_EPI8(x,y)       _mm_sub_epi8((x),(y))
+#define _MM_EXTRACT_EPI8(r,p)   _s2_extract_epi8((r),(p))
+
+#define _MM_MIN_EPU8(x,y)       _mm_min_epu8((x),(y))
+
+// Fill a SSE Register with 8 time the same 16bits integer value
+#define _MM_SET1_EPI16(x)       _mm_set1_epi16(x)
+
+#define _MM_INSERT_EPI16(r,x,i)	_mm_insert_epi16((r),(x),(i))
+#define _MM_CMPEQ_EPI16(x,y)    _mm_cmpeq_epi16((x),(y))
+#define _MM_CMPGT_EPI16(x,y)    _mm_cmpgt_epi16((x),(y))
+#define _MM_CMPGT_EPU16(x,y)    _mm_cmpgt_epu16((x),(y))    // n'existe pas ??
+#define _MM_CMPLT_EPI16(x,y)    _mm_cmplt_epi16((x),(y))
+#define _MM_MAX_EPI16(x,y)      _mm_max_epi16((x),(y))
+#define _MM_MIN_EPI16(x,y)      _mm_min_epi16((x),(y))
+#define _MM_ADD_EPI16(x,y)      _mm_add_epi16((x),(y))
+#define _MM_SUB_EPI16(x,y)      _mm_sub_epi16((x),(y))
+#define _MM_EXTRACT_EPI16(r,p)  _mm_extract_epi16((r),(p))
+#define _MM_UNPACKLO_EPI8(a,b)  _mm_unpacklo_epi8((a),(b))
+#define _MM_UNPACKHI_EPI8(a,b)  _mm_unpackhi_epi8((a),(b))
+#define _MM_ADDS_EPU16(x,y)     _mm_adds_epu16((x),(y))
+
+// Multiplication
+#define _MM_MULLO_EPI16(x,y)    _mm_mullo_epi16((x), (y))
+
+#define _MM_SRLI_EPI64(r,x)     _mm_srli_epi64((r),(x))
+#define _MM_SLLI_EPI64(r,x)     _mm_slli_epi64((r),(x))
+
+// Set a SSE Register to 0
+#define _MM_SETZERO_SI128() _mm_setzero_si128()
+
+#define _MM_AND_SI128(x,y)     _mm_and_si128((x),(y))
+#define _MM_ANDNOT_SI128(x,y)  _mm_andnot_si128((x),(y))
+#define _MM_OR_SI128(x,y)      _mm_or_si128((x),(y))
+#define _MM_XOR_SI128(x,y)     _mm_xor_si128((x),(y))
+#define _MM_SLLI_SI128(r,s)	   _mm_slli_si128((r),(s))
+#define _MM_SRLI_SI128(r,s)    _mm_srli_si128((r),(s))
+
+// Load a SSE register from an unaligned address
+#define _MM_LOADU_SI128(x) _mm_loadu_si128(x)
+
+// Load a SSE register from an aligned address (/!\ not defined when SSE not available)
+#define _MM_LOAD_SI128(x)  _mm_load_si128(x)
+
+// #define	_MM_UNPACKLO_EPI8(x,y) _mm_unpacklo_epi8((x),(y))
+
+#else /* __SSE2__ Not defined */
+
+static inline __m128i _em_set1_epi8(int x)
+{
+	um128 a;
+
+	x&=0xFF;
+	a.s8[0]=x;
+	a.s8[1]=x;
+	a.u16[1]=a.u16[0];
+	a.u32[1]=a.u32[0];
+	a.u64[1]=a.u64[0];
+
+	return a.i;
+}
+
+static inline __m128i _em_insert_epi8(__m128i r, int x, const int i)
+{
+	um128 a;
+	a.i=r;
+	a.s8[i]=x & 0xFF;
+	return a.i;
+}
+
+static inline __m128i _em_cmpeq_epi8(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s8[z]=(x.s8[z]==y.s8[z]) ? 0xFF:0
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+	R(8);
+	R(9);
+	R(10);
+	R(11);
+	R(12);
+	R(13);
+	R(14);
+	R(15);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_cmpgt_epi8(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s8[z]=(x.s8[z]>y.s8[z]) ? 0xFF:0
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+	R(8);
+	R(9);
+	R(10);
+	R(11);
+	R(12);
+	R(13);
+	R(14);
+	R(15);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_cmplt_epi8(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s8[z]=(x.s8[z]<y.s8[z]) ? 0xFF:0
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+	R(8);
+	R(9);
+	R(10);
+	R(11);
+	R(12);
+	R(13);
+	R(14);
+	R(15);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_max_epi8(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s8[z]=MAX(x.s8[z],y.s8[z])
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+	R(8);
+	R(9);
+	R(10);
+	R(11);
+	R(12);
+	R(13);
+	R(14);
+	R(15);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_min_epi8(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s8[z]=MIN(x.s8[z],y.s8[z])
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+	R(8);
+	R(9);
+	R(10);
+	R(11);
+	R(12);
+	R(13);
+	R(14);
+	R(15);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_add_epi8(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s8[z]=x.s8[z]+y.s8[z]
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+	R(8);
+	R(9);
+	R(10);
+	R(11);
+	R(12);
+	R(13);
+	R(14);
+	R(15);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_sub_epi8(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s8[z]=x.s8[z]+y.s8[z]
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+	R(8);
+	R(9);
+	R(10);
+	R(11);
+	R(12);
+	R(13);
+	R(14);
+	R(15);
+#undef R
+
+	return r.i;
+}
+
+
+static inline int _em_extract_epi8(__m128i r, const int i)
+{
+	um128 a;
+
+	a.i=r;
+
+	return a.s8[i] & 0xFF;
+}
+
+static inline __m128i _em_min_epu8(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.u8[z]=MIN(x.u8[z],y.u8[z])
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+	R(8);
+	R(9);
+	R(10);
+	R(11);
+	R(12);
+	R(13);
+	R(14);
+	R(15);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_set1_epi16(int x)
+{
+	um128 a;
+
+	x&=0xFFFF;
+	a.s16[0]=x;
+	a.s16[1]=x;
+	a.u32[1]=a.u32[0];
+	a.u64[1]=a.u64[0];
+
+	return a.i;
+}
+
+static inline __m128i _em_insert_epi16(__m128i r, int x, const int i)
+{
+	um128 a;
+	a.i=r;
+	a.s16[i]=x & 0xFFFF;
+	return a.i;
+}
+
+static inline __m128i _em_cmpeq_epi16(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s16[z]=(x.s16[z]==y.s16[z]) ? 0xFFFF:0
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_cmpgt_epi16(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s16[z]=(x.s16[z]>y.s16[z]) ? 0xFFFF:0
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_cmplt_epi16(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s16[z]=(x.s16[z]<y.s16[z]) ? 0xFFFF:0
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_max_epi16(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+
+#define R(z) 	r.s16[z]=MAX(x.s16[z],y.s16[z])
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_min_epi16(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+
+#define R(z) 	r.s16[z]=MIN(x.s16[z],y.s16[z])
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_add_epi16(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s16[z]=x.s16[z]+y.s16[z]
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_sub_epi16(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s16[z]=x.s16[z]+y.s16[z]
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+#undef R
+
+	return r.i;
+}
+
+static inline int _em_extract_epi16(__m128i r, const int i)
+{
+	um128 a;
+	a.i=r;
+	return a.s16[i] & 0xFFFF;
+}
+
+static inline __m128i _em_unpacklo_epi8(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s16[z]=(((int16_t)(y.s8[z])) << 8) | (int16_t)(x.s8[z])
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_unpackhi_epi8(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s16[z]=(((int16_t)(y.s8[z+8])) << 8) | (int16_t)(x.s8[z+8])
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_adds_epu16(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+
+#define R(z) 	r.u16[z]=x.u16[z]+y.u16[z]
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_srli_epi64(__m128i a, int b)
+{
+	um128 x;
+
+	x.i=a;
+
+	x.s64[0]>>=b;
+	x.s64[1]>>=b;
+
+	return x.i;
+}
+
+static inline __m128i _em_slli_epi64(__m128i a, int b)
+{
+	um128 x;
+
+	x.i=a;
+
+	x.s64[0]<<=b;
+	x.s64[1]<<=b;
+
+	return x.i;
+}
+
+static inline __m128i _em_setzero_si128()
+{
+	um128 x;
+
+	x.s64[0]=x.s64[1]=0;
+
+	return x.i;
+}
+
+static inline __m128i _em_and_si128(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+
+#define R(z) 	r.u64[z]=x.u64[z] & y.u64[z]
+	R(0);
+	R(1);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_andnot_si128(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+
+#define R(z) 	r.u64[z]=(~x.u64[z]) & y.u64[z]
+	R(0);
+	R(1);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_or_si128(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.u64[z]=x.u64[z] | y.u64[z]
+	R(0);
+	R(1);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_xor_si128(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.u64[z]=x.u64[z] ^ y.u64[z]
+	R(0);
+	R(1);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_slli_si128(__m128i a, int b)
+{
+	um128 x;
+
+	x.i=a;
+
+#define R(z) 	x.u8[z]=(z>=b) ? x.u8[z-b]:0
+	R(15);
+	R(14);
+	R(13);
+	R(12);
+	R(11);
+	R(10);
+	R(9);
+	R(8);
+	R(7);
+	R(6);
+	R(5);
+	R(4);
+	R(3);
+	R(2);
+	R(1);
+	R(0);
+#undef R
+
+	return x.i;
+}
+
+static inline __m128i _em_srli_si128(__m128i a, int b)
+{
+	um128 x;
+
+	x.i=a;
+
+#define R(z) 	x.u8[z]=((b+z) > 15) ? 0:x.u8[z+b]
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+	R(8);
+	R(9);
+	R(10);
+	R(11);
+	R(12);
+	R(13);
+	R(14);
+	R(15);
+#undef R
+
+	return x.i;
+}
+
+inline static __m128i _em_loadu_si128(__m128i const *P)
+{
+	um128 tmp;
+	um128 *pp=(um128*)P;
+
+	tmp.u8[0]=(*pp).u8[0];
+    tmp.u8[1]=(*pp).u8[1];
+	tmp.u8[2]=(*pp).u8[2];
+    tmp.u8[3]=(*pp).u8[3];
+	tmp.u8[4]=(*pp).u8[4];
+    tmp.u8[5]=(*pp).u8[5];
+	tmp.u8[6]=(*pp).u8[6];
+    tmp.u8[7]=(*pp).u8[7];
+	tmp.u8[8]=(*pp).u8[8];
+    tmp.u8[9]=(*pp).u8[9];
+	tmp.u8[10]=(*pp).u8[10];
+    tmp.u8[11]=(*pp).u8[11];
+	tmp.u8[12]=(*pp).u8[12];
+    tmp.u8[13]=(*pp).u8[13];
+	tmp.u8[14]=(*pp).u8[14];
+    tmp.u8[15]=(*pp).u8[15];
+	return tmp.i;
+}
+
+
+#define _MM_SET1_EPI8(x)        _em_set1_epi8(x)
+#define _MM_INSERT_EPI8(r,x,i)	_em_insert_epi8((r),(x),(i))
+#define _MM_CMPEQ_EPI8(x,y)     _em_cmpeq_epi8((x),(y))
+#define _MM_CMPGT_EPI8(x,y)     _em_cmpgt_epi8((x),(y))
+#define _MM_CMPLT_EPI8(x,y)     _em_cmplt_epi8((x),(y))
+#define _MM_MAX_EPI8(x,y)       _em_max_epi8((x),(y))
+#define _MM_MIN_EPI8(x,y)       _em_min_epi8((x),(y))
+#define _MM_ADD_EPI8(x,y)       _em_add_epi8((x),(y))
+#define _MM_SUB_EPI8(x,y)       _em_sub_epi8((x),(y))
+#define _MM_EXTRACT_EPI8(r,p)   _em_extract_epi8((r),(p))
+
+#define _MM_MIN_EPU8(x,y)       _em_min_epu8((x),(y))
+
+#define _MM_SET1_EPI16(x)       _em_set1_epi16(x)
+#define _MM_INSERT_EPI16(r,x,i)	_em_insert_epi16((r),(x),(i))
+#define _MM_CMPEQ_EPI16(x,y)    _em_cmpeq_epi16((x),(y))
+#define _MM_CMPGT_EPI16(x,y)    _em_cmpgt_epi16((x),(y))
+#define _MM_CMPLT_EPI16(x,y)    _em_cmplt_epi16((x),(y))
+#define _MM_MAX_EPI16(x,y)      _em_max_epi16((x),(y))
+#define _MM_MIN_EPI16(x,y)      _em_min_epi16((x),(y))
+#define _MM_ADD_EPI16(x,y)      _em_add_epi16((x),(y))
+#define _MM_SUB_EPI16(x,y)      _em_sub_epi16((x),(y))
+#define _MM_EXTRACT_EPI16(r,p)  _em_extract_epi16((r),(p))
+#define _MM_UNPACKLO_EPI8(a,b)  _em_unpacklo_epi8((a),(b))
+#define _MM_UNPACKHI_EPI8(a,b)  _em_unpackhi_epi8((a),(b))
+#define _MM_ADDS_EPU16(x,y)     _em_adds_epu16((x),(y))
+
+#define _MM_SRLI_EPI64(r,x)     _em_srli_epi64((r),(x))
+#define _MM_SLLI_EPI64(r,x)     _em_slli_epi64((r),(x))
+
+#define _MM_SETZERO_SI128()     _em_setzero_si128()
+
+#define _MM_AND_SI128(x,y)      _em_and_si128((x),(y))
+#define _MM_ANDNOT_SI128(x,y)   _em_andnot_si128((x),(y))
+#define _MM_OR_SI128(x,y)       _em_or_si128((x),(y))
+#define _MM_XOR_SI128(x,y)      _em_xor_si128((x),(y))
+#define _MM_SLLI_SI128(r,s)	    _em_slli_si128((r),(s))
+#define _MM_SRLI_SI128(r,s)	    _em_srli_si128((r),(s))
+
+#define _MM_LOADU_SI128(x)      _em_loadu_si128(x)
+#define _MM_LOAD_SI128(x)       _em_loadu_si128(x)
+
+
+#endif /* __SSE2__ */
+
+#define _MM_NOT_SI128(x)     _MM_XOR_SI128((x),(_MM_SET1_EPI8(0xFFFF)))
+
+#endif

src/bloom.c

@@ -0,0 +1,248 @@
+/*
+ *  Copyright (c) 2012-2015, Jyri J. Virkki
+ *  All rights reserved.
+ *
+ *  This file is under BSD license. See LICENSE file.
+ */
+
+/*
+ * Refer to bloom.h for documentation on the public interfaces.
+ */
+
+#include <assert.h>
+#include <fcntl.h>
+#include <math.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <unistd.h>
+
+#include "bloom.h"
+#include "murmurhash2.h"
+
+#define MAKESTRING(n) STRING(n)
+#define STRING(n) #n
+
+//#ifdef __linux__										// TODO commented because triggers error on luke21
+//unsigned detect_bucket_size(unsigned fallback_size);
+//#endif
+
+
+static int test_bit_set_bit(unsigned char * buf, unsigned int x, int set_bit)
+{
+  register uint32_t * word_buf = (uint32_t *)buf;
+  register unsigned int offset = x >> 5;
+  register uint32_t word = word_buf[offset];
+  register unsigned int mask = 1 << (x % 32);
+
+  if (word & mask) {
+    return 1;
+  } else {
+    if (set_bit) {
+      word_buf[offset] = word | mask;
+    }
+    return 0;
+  }
+}
+
+
+static int bloom_check_add(struct bloom * bloom,
+                           const void * buffer, int len, int add)
+{
+  if (bloom->ready == 0) {
+    (void)printf("bloom at %p not initialized!\n", (void *)bloom);
+    return -1;
+  }
+
+  int hits = 0;
+  register unsigned int a = murmurhash2(buffer, len, 0x9747b28c);
+  register unsigned int b = murmurhash2(buffer, len, a);
+  register unsigned int x;
+  register int i;	// TODO why was it unsigned?
+
+  unsigned bucket_index = (a % bloom->buckets);
+
+  unsigned char * bucket_ptr =
+    (bloom->bf + (bucket_index << bloom->bucket_bytes_exponent));
+
+  for (i = 0; i < bloom->hashes; i++) {
+    x = (a + i*b) & bloom->bucket_bits_fast_mod_operand;
+    if (test_bit_set_bit(bucket_ptr, x, add)) {
+      hits++;
+    }
+  }
+
+  if (hits == bloom->hashes) {
+    return 1;                // 1 == element already in (or collision)
+  }
+
+  return 0;
+}
+
+
+static void setup_buckets(struct bloom * bloom, unsigned int cache_size)
+{
+  // If caller passed a non-zero cache_size, use it as given, otherwise
+  // either compute it or use built-in default
+
+  if (cache_size == 0) {
+//#ifdef __linux__													// TODO commented because triggers error on luke21
+//    cache_size = detect_bucket_size(BLOOM_BUCKET_SIZE_FALLBACK);
+//#else
+    cache_size = BLOOM_BUCKET_SIZE_FALLBACK;
+//#endif
+  }
+
+  bloom->buckets = (bloom->bytes / cache_size);
+  bloom->bucket_bytes = cache_size;
+
+  // make sure bloom buffer bytes and bucket_bytes are even
+  int not_even_by = (bloom->bytes % bloom->bucket_bytes);
+
+  if (not_even_by) {
+    // adjust bytes
+    bloom->bytes += (bloom->bucket_bytes - not_even_by);
+    assert((bloom->bytes % bloom->bucket_bytes) == 0); // Should get even
+
+    // adjust bits
+    bloom->bits = bloom->bytes * 8;
+
+    // adjust bits per element
+    bloom->bpe = bloom->bits*1. / bloom->entries;
+
+    // adjust buckets
+    bloom->buckets++;
+  }
+
+  bloom->bucket_bytes_exponent = __builtin_ctz(cache_size);
+  bloom->bucket_bits_fast_mod_operand = (cache_size * 8 - 1);
+}
+
+
+int bloom_filter_size(int entries, double error)
+{
+	int bytes;
+	double num;
+	double denom;
+	double bpe;
+	int bits;
+	unsigned bucket_bytes;
+	int not_even_by;
+
+	num = log(error);
+	denom = 0.480453013918201; // ln(2)^2
+	bpe = -(num / denom);
+	bits = (int)(((double)entries) * bpe);
+
+	if (bits % 8) {
+		bytes = (bits / 8) + 1;
+	}
+	else {
+		bytes = bits / 8;
+	}
+
+	bucket_bytes = BLOOM_BUCKET_SIZE_FALLBACK;
+	not_even_by = bytes % bucket_bytes;
+	if (not_even_by) {
+	  // adjust bytes
+		bytes += (bucket_bytes - not_even_by);
+	}
+
+	return bytes;
+}
+
+
+int bloom_init_size(struct bloom * bloom, int entries, double error,
+                    unsigned int cache_size)
+{
+  bloom->ready = 0;
+
+  if (entries < 1 || error == 0) {
+    return 1;
+  }
+
+  bloom->entries = entries;
+  bloom->error = error;
+
+  double num = log(bloom->error);
+  double denom = 0.480453013918201; // ln(2)^2
+  bloom->bpe = -(num / denom);
+
+  double dentries = (double)entries;
+  bloom->bits = (int)(dentries * bloom->bpe);
+
+  if (bloom->bits % 8) {
+    bloom->bytes = (bloom->bits / 8) + 1;
+  } else {
+    bloom->bytes = bloom->bits / 8;
+  }
+
+  bloom->hashes = (int)ceil(0.693147180559945 * bloom->bpe);  // ln(2)
+
+  setup_buckets(bloom, cache_size);
+
+  // celine.mercier@metabarcoding.org :
+  // Replaced the calloc with a memset, as the memory for the bloom filter is mapped in our data structure
+  memset(bloom->bf, 0, bloom->bytes);
+  //bloom->bf = (unsigned char *)calloc(bloom->bytes, sizeof(unsigned char));
+  //if (bloom->bf == NULL) {
+  //  return 1;
+  //}
+
+  bloom->ready = 1;
+  return 0;
+}
+
+
+int bloom_init(struct bloom * bloom, int entries)  //, double error)
+{
+  return bloom_init_size(bloom, entries, BLOOM_FILTER_ERROR_RATE, 0);
+}
+
+
+int bloom_check(struct bloom * bloom, const void * buffer, int len)
+{
+  return bloom_check_add(bloom, buffer, len, 0);
+}
+
+
+int bloom_add(struct bloom * bloom, const void * buffer, int len)
+{
+  return bloom_check_add(bloom, buffer, len, 1);
+}
+
+
+void bloom_print(struct bloom * bloom)
+{
+  (void)printf("bloom at %p\n", (void *)bloom);
+  (void)printf(" ->entries = %d\n", bloom->entries);
+  (void)printf(" ->error = %f\n", bloom->error);
+  (void)printf(" ->bits = %d\n", bloom->bits);
+  (void)printf(" ->bits per elem = %f\n", bloom->bpe);
+  (void)printf(" ->bytes = %d\n", bloom->bytes);
+  (void)printf(" ->buckets = %u\n", bloom->buckets);
+  (void)printf(" ->bucket_bytes = %u\n", bloom->bucket_bytes);
+  (void)printf(" ->bucket_bytes_exponent = %u\n",
+               bloom->bucket_bytes_exponent);
+  (void)printf(" ->bucket_bits_fast_mod_operand = 0%o\n",
+               bloom->bucket_bits_fast_mod_operand);
+  (void)printf(" ->hash functions = %d\n", bloom->hashes);
+}
+
+
+void bloom_free(struct bloom * bloom)
+{
+  if (bloom->ready) {
+    free(bloom->bf);
+  }
+  bloom->ready = 0;
+}
+
+
+const char * bloom_version()
+{
+  return MAKESTRING(BLOOM_VERSION);
+}

src/bloom.h

@@ -0,0 +1,210 @@
+/*
+ *  Copyright (c) 2012-2015, Jyri J. Virkki
+ *  All rights reserved.
+ *
+ *  This file is under BSD license. See LICENSE file.
+ */
+
+#ifndef _BLOOM_H
+#define _BLOOM_H
+
+
+/** ***************************************************************************
+ * Added by celine.mercier@metabarcoding.org
+ *
+ * Bloom filter error rate wanted.
+ *
+ */
+#define BLOOM_FILTER_ERROR_RATE (0.001)
+
+
+/** ***************************************************************************
+ * On Linux, the code attempts to compute a bucket size based on CPU cache
+ * size info, if available. If that fails for any reason, this fallback size
+ * is used instead.
+ *
+ * On non-Linux systems, this is the bucket size always used unless the
+ * caller overrides it (see bloom_init_size()).
+ *
+ */
+#define BLOOM_BUCKET_SIZE_FALLBACK (32 * 1024)
+
+
+/** ***************************************************************************
+ * It was found that using multiplier x0.5 for CPU L1 cache size is
+ * more effective in terms of CPU usage and, surprisingly, collisions
+ * number.
+ *
+ * Feel free to tune this constant the way it will work for you.
+ *
+ */
+#define BLOOM_L1_CACHE_SIZE_DIV 1
+
+
+/** ***************************************************************************
+ * Structure to keep track of one bloom filter.  Caller needs to
+ * allocate this and pass it to the functions below. First call for
+ * every struct must be to bloom_init().
+ *
+ */
+struct bloom
+{
+  // These fields are part of the public interface of this structure.
+  // Client code may read these values if desired. Client code MUST NOT
+  // modify any of these.
+  int entries;
+  double error;
+  int bits;
+  int bytes;
+  int hashes;
+
+  // Fields below are private to the implementation. These may go away or
+  // change incompatibly at any moment. Client code MUST NOT access or rely
+  // on these.
+  unsigned buckets;
+  unsigned bucket_bytes;
+
+  // x86 CPU divide by/multiply by operation optimization helpers
+  unsigned bucket_bytes_exponent;
+  unsigned bucket_bits_fast_mod_operand;
+
+  double bpe;
+  int ready;
+
+  unsigned char bf[];
+};
+
+typedef struct bloom bloom_t;
+
+
+/** ***************************************************************************
+ * Added by celine.mercier@metabarcoding.org
+ *
+ * This function computes the size needed by the bloom filter
+ * depending on the number of entries and the error rate.
+ *
+ */
+int bloom_filter_size(int entries, double error);
+
+
+/** ***************************************************************************
+ * Initialize the bloom filter for use.
+ *
+ * The filter is initialized with a bit field and number of hash functions
+ * according to the computations from the wikipedia entry:
+ *     http://en.wikipedia.org/wiki/Bloom_filter
+ *
+ * Optimal number of bits is:
+ *     bits = (entries * ln(error)) / ln(2)^2
+ *
+ * Optimal number of hash functions is:
+ *     hashes = bpe * ln(2)
+ *
+ * Parameters:
+ * -----------
+ *     bloom   - Pointer to an allocated struct bloom (see above).
+ *     entries - The expected number of entries which will be inserted.
+ *     error   - Probability of collision (as long as entries are not
+ *               exceeded).
+ *
+ * Return:
+ * -------
+ *     0 - on success
+ *     1 - on failure
+ *
+ */
+int bloom_init(struct bloom * bloom, int entries); //, double error);
+
+
+/** ***************************************************************************
+ * Initialize the bloom filter for use.
+ *
+ * See comments above for general information.
+ *
+ * This is the same as bloom_init() but allows the caller to pass in a
+ * cache_size to override the internal value (which is either computed
+ * or the default of BLOOM_BUCKET_SIZE_FALLBACK). Mostly useful for
+ * experimenting.
+ *
+ * See misc/bucketsize for a script which can help identify a good value
+ * for cache_size.
+ *
+ */
+int bloom_init_size(struct bloom * bloom, int entries, double error,
+                    unsigned int cache_size);
+
+
+/** ***************************************************************************
+ * Check if the given element is in the bloom filter. Remember this may
+ * return false positive if a collision occured.
+ *
+ * Parameters:
+ * -----------
+ *     bloom  - Pointer to an allocated struct bloom (see above).
+ *     buffer - Pointer to buffer containing element to check.
+ *     len    - Size of 'buffer'.
+ *
+ * Return:
+ * -------
+ *     0 - element is not present
+ *     1 - element is present (or false positive due to collision)
+ *    -1 - bloom not initialized
+ *
+ */
+int bloom_check(struct bloom * bloom, const void * buffer, int len);
+
+
+/** ***************************************************************************
+ * Add the given element to the bloom filter.
+ * The return code indicates if the element (or a collision) was already in,
+ * so for the common check+add use case, no need to call check separately.
+ *
+ * Parameters:
+ * -----------
+ *     bloom  - Pointer to an allocated struct bloom (see above).
+ *     buffer - Pointer to buffer containing element to add.
+ *     len    - Size of 'buffer'.
+ *
+ * Return:
+ * -------
+ *     0 - element was not present and was added
+ *     1 - element (or a collision) had already been added previously
+ *    -1 - bloom not initialized
+ *
+ */
+int bloom_add(struct bloom * bloom, const void * buffer, int len);
+
+
+/** ***************************************************************************
+ * Print (to stdout) info about this bloom filter. Debugging aid.
+ *
+ */
+void bloom_print(struct bloom * bloom);
+
+
+/** ***************************************************************************
+ * Deallocate internal storage.
+ *
+ * Upon return, the bloom struct is no longer usable. You may call bloom_init
+ * again on the same struct to reinitialize it again.
+ *
+ * Parameters:
+ * -----------
+ *     bloom  - Pointer to an allocated struct bloom (see above).
+ *
+ * Return: none
+ *
+ */
+void bloom_free(struct bloom * bloom);
+
+
+/** ***************************************************************************
+ * Returns version string compiled into library.
+ *
+ * Return: version string
+ *
+ */
+const char * bloom_version();
+
+
+#endif

src/char_str_indexer.c

@@ -0,0 +1,73 @@
+/****************************************************************************
+ * Character string indexing functions                                      *
+ ****************************************************************************/
+
+/**
+ * @file char_str_indexer.c
+ * @author Celine Mercier
+ * @date April 12th 2016
+ * @brief Functions handling the indexing and retrieval of character strings.
+ */
+
+
+#include <stdlib.h>
+#include <stdio.h>
+#include <math.h>
+
+#include "char_str_indexer.h"
+#include "obiblob.h"
+#include "obiblob_indexer.h"
+#include "obidebug.h"
+#include "obitypes.h"
+
+
+#define DEBUG_LEVEL 0	// TODO has to be defined somewhere else (cython compil flag?)
+
+
+Obi_blob_p obi_str_to_blob(const char* value)
+{
+	int32_t length;
+
+	// Compute the number of bytes on which the value will be encoded
+	length = strlen(value) + 1;		// +1 to store \0 at the end (makes retrieving faster)
+
+	return obi_blob((byte_t*)value, ELEMENT_SIZE_STR, length, length);
+}
+
+
+const char* obi_blob_to_str(Obi_blob_p value_b)
+{
+	return value_b->value;
+}
+
+
+index_t obi_index_char_str(Obi_indexer_p indexer, const char* value)
+{
+	Obi_blob_p  value_b;
+	index_t 	idx;
+
+	// Encode value
+	value_b = obi_str_to_blob(value);
+	if (value_b == NULL)
+		return -1;
+
+	// Add in the indexer
+	idx = obi_indexer_add(indexer, value_b);
+
+	free(value_b);
+
+	return idx;
+}
+
+
+const char* obi_retrieve_char_str(Obi_indexer_p indexer, index_t idx)
+{
+	Obi_blob_p  value_b;
+
+	// Get encoded value
+	value_b = obi_indexer_get(indexer, idx);
+
+	// Return decoded character string
+	return obi_blob_to_str(value_b);
+}
+

src/char_str_indexer.h

@@ -0,0 +1,87 @@
+/****************************************************************************
+ * DNA sequence indexer header file	                                        *
+ ****************************************************************************/
+
+/**
+ * @file dna_seq_indexer.h
+ * @author Celine Mercier
+ * @date April 12th 2016
+ * @brief Header file for the functions handling the indexing of DNA sequences.
+ */
+
+
+#ifndef CHAR_STR_INDEXER_H_
+#define CHAR_STR_INDEXER_H_
+
+
+#include <stdlib.h>
+#include <stdio.h>
+
+#include "obitypes.h"
+#include "obiblob.h"
+#include "obiblob_indexer.h"
+
+
+/**
+ * @brief Converts a character string to a blob.
+ *
+ * @warning The blob must be freed by the caller.
+ *
+ * @param value The character string to convert.
+ *
+ * @returns A pointer to the blob created.
+ * @retval NULL if an error occurred.
+ *
+ * @since October 2015
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+Obi_blob_p obi_str_to_blob(const char* value);
+
+
+/**
+ * @brief Converts a blob to a character string.
+ *
+ * @warning The character string returned is mapped.
+ *
+ * @param value_b The blob to convert.
+ *
+ * @returns A pointer on the character string contained in the blob.
+ *
+ * @since October 2015
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+const char* obi_blob_to_str(Obi_blob_p value_b);
+
+
+/**
+ * @brief Stores a character string in an indexer and returns the index.
+ *
+ * @param indexer The indexer structure.
+ * @param value The character string to index.
+ *
+ * @returns The index referring to the stored character string in the indexer.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+index_t obi_index_char_str(Obi_indexer_p indexer, const char* value);
+
+
+/**
+ * @brief Retrieves a character string from an indexer.
+ *
+ * @warning The character string returned is mapped.
+ *
+ * @param indexer The indexer structure.
+ * @param idx The index referring to the character string to retrieve in the indexer.
+ *
+ * @returns A pointer on the character string.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+const char* obi_retrieve_char_str(Obi_indexer_p indexer, index_t idx);
+
+
+#endif /* CHAR_STR_INDEXER_H_ */
+

src/crc64.c

@@ -0,0 +1,198 @@
+/* Redis uses the CRC64 variant with "Jones" coefficients and init value of 0.
+ *
+ * Specification of this CRC64 variant follows:
+ * Name: crc-64-jones
+ * Width: 64 bites
+ * Poly: 0xad93d23594c935a9
+ * Reflected In: True
+ * Xor_In: 0xffffffffffffffff
+ * Reflected_Out: True
+ * Xor_Out: 0x0
+ * Check("123456789"): 0xe9c6d914c4b8d9ca
+ *
+ * Copyright (c) 2012, Salvatore Sanfilippo <antirez at gmail dot com>
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE. */
+
+#include <stdint.h>
+
+static const uint64_t crc64_tab[256] = {
+    UINT64_C(0x0000000000000000), UINT64_C(0x7ad870c830358979),
+    UINT64_C(0xf5b0e190606b12f2), UINT64_C(0x8f689158505e9b8b),
+    UINT64_C(0xc038e5739841b68f), UINT64_C(0xbae095bba8743ff6),
+    UINT64_C(0x358804e3f82aa47d), UINT64_C(0x4f50742bc81f2d04),
+    UINT64_C(0xab28ecb46814fe75), UINT64_C(0xd1f09c7c5821770c),
+    UINT64_C(0x5e980d24087fec87), UINT64_C(0x24407dec384a65fe),
+    UINT64_C(0x6b1009c7f05548fa), UINT64_C(0x11c8790fc060c183),
+    UINT64_C(0x9ea0e857903e5a08), UINT64_C(0xe478989fa00bd371),
+    UINT64_C(0x7d08ff3b88be6f81), UINT64_C(0x07d08ff3b88be6f8),
+    UINT64_C(0x88b81eabe8d57d73), UINT64_C(0xf2606e63d8e0f40a),
+    UINT64_C(0xbd301a4810ffd90e), UINT64_C(0xc7e86a8020ca5077),
+    UINT64_C(0x4880fbd87094cbfc), UINT64_C(0x32588b1040a14285),
+    UINT64_C(0xd620138fe0aa91f4), UINT64_C(0xacf86347d09f188d),
+    UINT64_C(0x2390f21f80c18306), UINT64_C(0x594882d7b0f40a7f),
+    UINT64_C(0x1618f6fc78eb277b), UINT64_C(0x6cc0863448deae02),
+    UINT64_C(0xe3a8176c18803589), UINT64_C(0x997067a428b5bcf0),
+    UINT64_C(0xfa11fe77117cdf02), UINT64_C(0x80c98ebf2149567b),
+    UINT64_C(0x0fa11fe77117cdf0), UINT64_C(0x75796f2f41224489),
+    UINT64_C(0x3a291b04893d698d), UINT64_C(0x40f16bccb908e0f4),
+    UINT64_C(0xcf99fa94e9567b7f), UINT64_C(0xb5418a5cd963f206),
+    UINT64_C(0x513912c379682177), UINT64_C(0x2be1620b495da80e),
+    UINT64_C(0xa489f35319033385), UINT64_C(0xde51839b2936bafc),
+    UINT64_C(0x9101f7b0e12997f8), UINT64_C(0xebd98778d11c1e81),
+    UINT64_C(0x64b116208142850a), UINT64_C(0x1e6966e8b1770c73),
+    UINT64_C(0x8719014c99c2b083), UINT64_C(0xfdc17184a9f739fa),
+    UINT64_C(0x72a9e0dcf9a9a271), UINT64_C(0x08719014c99c2b08),
+    UINT64_C(0x4721e43f0183060c), UINT64_C(0x3df994f731b68f75),
+    UINT64_C(0xb29105af61e814fe), UINT64_C(0xc849756751dd9d87),
+    UINT64_C(0x2c31edf8f1d64ef6), UINT64_C(0x56e99d30c1e3c78f),
+    UINT64_C(0xd9810c6891bd5c04), UINT64_C(0xa3597ca0a188d57d),
+    UINT64_C(0xec09088b6997f879), UINT64_C(0x96d1784359a27100),
+    UINT64_C(0x19b9e91b09fcea8b), UINT64_C(0x636199d339c963f2),
+    UINT64_C(0xdf7adabd7a6e2d6f), UINT64_C(0xa5a2aa754a5ba416),
+    UINT64_C(0x2aca3b2d1a053f9d), UINT64_C(0x50124be52a30b6e4),
+    UINT64_C(0x1f423fcee22f9be0), UINT64_C(0x659a4f06d21a1299),
+    UINT64_C(0xeaf2de5e82448912), UINT64_C(0x902aae96b271006b),
+    UINT64_C(0x74523609127ad31a), UINT64_C(0x0e8a46c1224f5a63),
+    UINT64_C(0x81e2d7997211c1e8), UINT64_C(0xfb3aa75142244891),
+    UINT64_C(0xb46ad37a8a3b6595), UINT64_C(0xceb2a3b2ba0eecec),
+    UINT64_C(0x41da32eaea507767), UINT64_C(0x3b024222da65fe1e),
+    UINT64_C(0xa2722586f2d042ee), UINT64_C(0xd8aa554ec2e5cb97),
+    UINT64_C(0x57c2c41692bb501c), UINT64_C(0x2d1ab4dea28ed965),
+    UINT64_C(0x624ac0f56a91f461), UINT64_C(0x1892b03d5aa47d18),
+    UINT64_C(0x97fa21650afae693), UINT64_C(0xed2251ad3acf6fea),
+    UINT64_C(0x095ac9329ac4bc9b), UINT64_C(0x7382b9faaaf135e2),
+    UINT64_C(0xfcea28a2faafae69), UINT64_C(0x8632586aca9a2710),
+    UINT64_C(0xc9622c4102850a14), UINT64_C(0xb3ba5c8932b0836d),
+    UINT64_C(0x3cd2cdd162ee18e6), UINT64_C(0x460abd1952db919f),
+    UINT64_C(0x256b24ca6b12f26d), UINT64_C(0x5fb354025b277b14),
+    UINT64_C(0xd0dbc55a0b79e09f), UINT64_C(0xaa03b5923b4c69e6),
+    UINT64_C(0xe553c1b9f35344e2), UINT64_C(0x9f8bb171c366cd9b),
+    UINT64_C(0x10e3202993385610), UINT64_C(0x6a3b50e1a30ddf69),
+    UINT64_C(0x8e43c87e03060c18), UINT64_C(0xf49bb8b633338561),
+    UINT64_C(0x7bf329ee636d1eea), UINT64_C(0x012b592653589793),
+    UINT64_C(0x4e7b2d0d9b47ba97), UINT64_C(0x34a35dc5ab7233ee),
+    UINT64_C(0xbbcbcc9dfb2ca865), UINT64_C(0xc113bc55cb19211c),
+    UINT64_C(0x5863dbf1e3ac9dec), UINT64_C(0x22bbab39d3991495),
+    UINT64_C(0xadd33a6183c78f1e), UINT64_C(0xd70b4aa9b3f20667),
+    UINT64_C(0x985b3e827bed2b63), UINT64_C(0xe2834e4a4bd8a21a),
+    UINT64_C(0x6debdf121b863991), UINT64_C(0x1733afda2bb3b0e8),
+    UINT64_C(0xf34b37458bb86399), UINT64_C(0x8993478dbb8deae0),
+    UINT64_C(0x06fbd6d5ebd3716b), UINT64_C(0x7c23a61ddbe6f812),
+    UINT64_C(0x3373d23613f9d516), UINT64_C(0x49aba2fe23cc5c6f),
+    UINT64_C(0xc6c333a67392c7e4), UINT64_C(0xbc1b436e43a74e9d),
+    UINT64_C(0x95ac9329ac4bc9b5), UINT64_C(0xef74e3e19c7e40cc),
+    UINT64_C(0x601c72b9cc20db47), UINT64_C(0x1ac40271fc15523e),
+    UINT64_C(0x5594765a340a7f3a), UINT64_C(0x2f4c0692043ff643),
+    UINT64_C(0xa02497ca54616dc8), UINT64_C(0xdafce7026454e4b1),
+    UINT64_C(0x3e847f9dc45f37c0), UINT64_C(0x445c0f55f46abeb9),
+    UINT64_C(0xcb349e0da4342532), UINT64_C(0xb1eceec59401ac4b),
+    UINT64_C(0xfebc9aee5c1e814f), UINT64_C(0x8464ea266c2b0836),
+    UINT64_C(0x0b0c7b7e3c7593bd), UINT64_C(0x71d40bb60c401ac4),
+    UINT64_C(0xe8a46c1224f5a634), UINT64_C(0x927c1cda14c02f4d),
+    UINT64_C(0x1d148d82449eb4c6), UINT64_C(0x67ccfd4a74ab3dbf),
+    UINT64_C(0x289c8961bcb410bb), UINT64_C(0x5244f9a98c8199c2),
+    UINT64_C(0xdd2c68f1dcdf0249), UINT64_C(0xa7f41839ecea8b30),
+    UINT64_C(0x438c80a64ce15841), UINT64_C(0x3954f06e7cd4d138),
+    UINT64_C(0xb63c61362c8a4ab3), UINT64_C(0xcce411fe1cbfc3ca),
+    UINT64_C(0x83b465d5d4a0eece), UINT64_C(0xf96c151de49567b7),
+    UINT64_C(0x76048445b4cbfc3c), UINT64_C(0x0cdcf48d84fe7545),
+    UINT64_C(0x6fbd6d5ebd3716b7), UINT64_C(0x15651d968d029fce),
+    UINT64_C(0x9a0d8ccedd5c0445), UINT64_C(0xe0d5fc06ed698d3c),
+    UINT64_C(0xaf85882d2576a038), UINT64_C(0xd55df8e515432941),
+    UINT64_C(0x5a3569bd451db2ca), UINT64_C(0x20ed197575283bb3),
+    UINT64_C(0xc49581ead523e8c2), UINT64_C(0xbe4df122e51661bb),
+    UINT64_C(0x3125607ab548fa30), UINT64_C(0x4bfd10b2857d7349),
+    UINT64_C(0x04ad64994d625e4d), UINT64_C(0x7e7514517d57d734),
+    UINT64_C(0xf11d85092d094cbf), UINT64_C(0x8bc5f5c11d3cc5c6),
+    UINT64_C(0x12b5926535897936), UINT64_C(0x686de2ad05bcf04f),
+    UINT64_C(0xe70573f555e26bc4), UINT64_C(0x9ddd033d65d7e2bd),
+    UINT64_C(0xd28d7716adc8cfb9), UINT64_C(0xa85507de9dfd46c0),
+    UINT64_C(0x273d9686cda3dd4b), UINT64_C(0x5de5e64efd965432),
+    UINT64_C(0xb99d7ed15d9d8743), UINT64_C(0xc3450e196da80e3a),
+    UINT64_C(0x4c2d9f413df695b1), UINT64_C(0x36f5ef890dc31cc8),
+    UINT64_C(0x79a59ba2c5dc31cc), UINT64_C(0x037deb6af5e9b8b5),
+    UINT64_C(0x8c157a32a5b7233e), UINT64_C(0xf6cd0afa9582aa47),
+    UINT64_C(0x4ad64994d625e4da), UINT64_C(0x300e395ce6106da3),
+    UINT64_C(0xbf66a804b64ef628), UINT64_C(0xc5bed8cc867b7f51),
+    UINT64_C(0x8aeeace74e645255), UINT64_C(0xf036dc2f7e51db2c),
+    UINT64_C(0x7f5e4d772e0f40a7), UINT64_C(0x05863dbf1e3ac9de),
+    UINT64_C(0xe1fea520be311aaf), UINT64_C(0x9b26d5e88e0493d6),
+    UINT64_C(0x144e44b0de5a085d), UINT64_C(0x6e963478ee6f8124),
+    UINT64_C(0x21c640532670ac20), UINT64_C(0x5b1e309b16452559),
+    UINT64_C(0xd476a1c3461bbed2), UINT64_C(0xaeaed10b762e37ab),
+    UINT64_C(0x37deb6af5e9b8b5b), UINT64_C(0x4d06c6676eae0222),
+    UINT64_C(0xc26e573f3ef099a9), UINT64_C(0xb8b627f70ec510d0),
+    UINT64_C(0xf7e653dcc6da3dd4), UINT64_C(0x8d3e2314f6efb4ad),
+    UINT64_C(0x0256b24ca6b12f26), UINT64_C(0x788ec2849684a65f),
+    UINT64_C(0x9cf65a1b368f752e), UINT64_C(0xe62e2ad306bafc57),
+    UINT64_C(0x6946bb8b56e467dc), UINT64_C(0x139ecb4366d1eea5),
+    UINT64_C(0x5ccebf68aecec3a1), UINT64_C(0x2616cfa09efb4ad8),
+    UINT64_C(0xa97e5ef8cea5d153), UINT64_C(0xd3a62e30fe90582a),
+    UINT64_C(0xb0c7b7e3c7593bd8), UINT64_C(0xca1fc72bf76cb2a1),
+    UINT64_C(0x45775673a732292a), UINT64_C(0x3faf26bb9707a053),
+    UINT64_C(0x70ff52905f188d57), UINT64_C(0x0a2722586f2d042e),
+    UINT64_C(0x854fb3003f739fa5), UINT64_C(0xff97c3c80f4616dc),
+    UINT64_C(0x1bef5b57af4dc5ad), UINT64_C(0x61372b9f9f784cd4),
+    UINT64_C(0xee5fbac7cf26d75f), UINT64_C(0x9487ca0fff135e26),
+    UINT64_C(0xdbd7be24370c7322), UINT64_C(0xa10fceec0739fa5b),
+    UINT64_C(0x2e675fb4576761d0), UINT64_C(0x54bf2f7c6752e8a9),
+    UINT64_C(0xcdcf48d84fe75459), UINT64_C(0xb71738107fd2dd20),
+    UINT64_C(0x387fa9482f8c46ab), UINT64_C(0x42a7d9801fb9cfd2),
+    UINT64_C(0x0df7adabd7a6e2d6), UINT64_C(0x772fdd63e7936baf),
+    UINT64_C(0xf8474c3bb7cdf024), UINT64_C(0x829f3cf387f8795d),
+    UINT64_C(0x66e7a46c27f3aa2c), UINT64_C(0x1c3fd4a417c62355),
+    UINT64_C(0x935745fc4798b8de), UINT64_C(0xe98f353477ad31a7),
+    UINT64_C(0xa6df411fbfb21ca3), UINT64_C(0xdc0731d78f8795da),
+    UINT64_C(0x536fa08fdfd90e51), UINT64_C(0x29b7d047efec8728),
+};
+
+
+uint64_t crc64(const char* s, uint64_t l)
+{
+	uint64_t j;
+    uint64_t crc = 0;
+
+    for (j = 0; j < l; j++)
+    {
+        uint8_t byte = s[j];
+        crc = crc64_tab[(uint8_t)crc ^ byte] ^ (crc >> 8);
+    }
+    return crc;
+}
+
+
+/* Test main */
+//#ifdef TEST_MAIN
+//#include <stdio.h>
+//int main(void) {
+//    printf("e9c6d914c4b8d9ca == %016llx\n",
+//        (unsigned long long) crc64(0,(unsigned char*)"123456789",9));
+//    return 0;
+//}
+//#endif
+
+

src/crc64.h

@@ -0,0 +1,14 @@
+/**
+ * @file crc64.h
+ * @date March 24th 2016
+ * @brief Header file for CRC64 function.
+ */
+
+#include <stdint.h>
+
+
+/**
+ * @brief Generates and returns a 64-bit Cyclic Redundancy Check from a
+ *        character string s of length l.
+ */
+uint64_t crc64(const char* s, uint64_t l);

src/dna_seq_indexer.c

@@ -0,0 +1,103 @@
+/****************************************************************************
+ * DNA sequence indexing functions                                          *
+ ****************************************************************************/
+
+/**
+ * @file dna_seq_indexer.c
+ * @author Celine Mercier
+ * @date April 12th 2016
+ * @brief Functions handling the indexing and retrieval of DNA sequences.
+ */
+
+
+#include <stdlib.h>
+#include <stdio.h>
+#include <math.h>
+
+#include "dna_seq_indexer.h"
+#include "obiblob.h"
+#include "obiblob_indexer.h"
+#include "obidebug.h"
+#include "obitypes.h"
+
+
+#define DEBUG_LEVEL 0	// TODO has to be defined somewhere else (cython compil flag?)
+
+
+Obi_blob_p obi_seq_to_blob(const char* seq)
+{
+	Obi_blob_p 		 value_b;
+	int32_t          length_encoded_seq;	// length of the encoded sequence in bytes
+	int32_t          seq_length;
+	byte_t*          encoded_seq;
+
+	seq_length = strlen(seq);
+
+	// Check if just ATGC and encode accordingly
+	if (only_ATGC(seq))
+	{
+		// Compute the length (in bytes) of the encoded sequence
+		length_encoded_seq = ceil((double) seq_length / (double) 4.0);
+		// Encode
+		encoded_seq = encode_seq_on_2_bits(seq, seq_length);
+		if (encoded_seq == NULL)
+			return NULL;
+		value_b = obi_blob(encoded_seq, ELEMENT_SIZE_SEQ_2, length_encoded_seq, seq_length);
+	}
+	else
+	{
+		// Compute the length (in bytes) of the encoded sequence
+		length_encoded_seq = ceil((double) seq_length / (double) 2.0);
+		// Encode
+		encoded_seq = encode_seq_on_4_bits(seq, seq_length);
+		if (encoded_seq == NULL)
+			return NULL;
+		value_b = obi_blob(encoded_seq, ELEMENT_SIZE_SEQ_4, length_encoded_seq, seq_length);
+	}
+
+	free(encoded_seq);
+
+	return value_b;
+}
+
+
+char* obi_blob_to_seq(Obi_blob_p value_b)
+{
+	// Decode
+	if (value_b->element_size == 2)
+		return decode_seq_on_2_bits(value_b->value, value_b->length_decoded_value);
+	else
+		return decode_seq_on_4_bits(value_b->value, value_b->length_decoded_value);
+}
+
+
+index_t obi_index_dna_seq(Obi_indexer_p indexer, const char* value)
+{
+	Obi_blob_p  value_b;
+	index_t 	idx;
+
+	// Encode value
+	value_b = obi_seq_to_blob(value);
+	if (value_b == NULL)
+		return -1;
+
+	// Add in the indexer
+	idx = obi_indexer_add(indexer, value_b);
+
+	free(value_b);
+
+	return idx;
+}
+
+
+char* obi_retrieve_dna_seq(Obi_indexer_p indexer, index_t idx)
+{
+	Obi_blob_p  value_b;
+
+	// Get encoded value
+	value_b = obi_indexer_get(indexer, idx);
+
+	// Return decoded sequence
+	return obi_blob_to_seq(value_b);
+}
+

src/dna_seq_indexer.h

@@ -0,0 +1,87 @@
+/****************************************************************************
+ * DNA sequence indexer header file	                                        *
+ ****************************************************************************/
+
+/**
+ * @file dna_seq_indexer.h
+ * @author Celine Mercier
+ * @date April 12th 2016
+ * @brief Header file for the functions handling the indexing of DNA sequences.
+ */
+
+
+#ifndef DNA_SEQ_INDEXER_H_
+#define DNA_SEQ_INDEXER_H_
+
+
+#include <stdlib.h>
+#include <stdio.h>
+
+#include "obidms.h"
+#include "obitypes.h"
+#include "obiblob.h"
+#include "obiblob_indexer.h"
+
+
+/**
+ * @brief Converts a DNA sequence to a blob.
+ *
+ * @warning The blob must be freed by the caller.
+ *
+ * @param value The DNA sequence to convert.
+ *
+ * @returns A pointer to the blob created.
+ * @retval NULL if an error occurred.
+ *
+ * @since November 2015
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+Obi_blob_p obi_seq_to_blob(const char* seq);
+
+
+/**
+ * @brief Converts a blob to a DNA sequence.
+ *
+ * @param value_b The blob to convert.
+ *
+ * @returns A pointer to the DNA sequence contained in the blob.
+ * @retval NULL if an error occurred.
+ *
+ * @since November 2015
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+char* obi_blob_to_seq(Obi_blob_p value_b);
+
+
+/**
+ * @brief Stores a DNA sequence in an indexer and returns the index.
+ *
+ * @param indexer The indexer structure.
+ * @param value The DNA sequence to index.
+ *
+ * @returns The index referring to the stored DNA sequence in the indexer.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+index_t obi_index_dna_seq(Obi_indexer_p indexer, const char* value);
+
+
+/**
+ * @brief Retrieves a DNA sequence from an indexer.
+ *
+ * @warning The DNA sequence returned must be freed by the caller.
+ *
+ * @param indexer The indexer structure.
+ * @param idx The index referring to the DNA sequence to retrieve in the indexer.
+ *
+ * @returns A pointer on the DNA sequence.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+char* obi_retrieve_dna_seq(Obi_indexer_p indexer, index_t idx);
+
+
+#endif /* DNA_SEQ_INDEXER_H_ */
+

src/encode.c

@@ -0,0 +1,367 @@
+/****************************************************************************
+ * Encoding functions                                                       *
+ ****************************************************************************/
+
+/**
+ * @file encode.c
+ * @author Celine Mercier
+ * @date November 18th 2015
+ * @brief Functions encoding DNA sequences.
+ */
+
+
+#include <stdlib.h>
+#include <stdio.h>
+#include <stdint.h>
+#include <math.h>
+
+#include "encode.h"
+#include "obierrno.h"
+#include "obitypes.h"	// For byte_t type
+#include "obidebug.h"
+
+
+#define DEBUG_LEVEL 0	// TODO has to be defined somewhere else (cython compil flag?)
+
+
+// TODO: endianness problem?
+
+
+
+bool only_ATGC(const char* seq)
+{
+	const char* c = seq;
+
+	while (*c)
+	{
+		if (!((*c == 'A') || \
+			  (*c == 'T') || \
+			  (*c == 'G') || \
+			  (*c == 'C') || \
+			  (*c == 'a') || \
+			  (*c == 't') || \
+			  (*c == 'g') || \
+			  (*c == 'c')))
+		{
+			return 0;
+		}
+		else
+		{
+			c++;
+		}
+	}
+	return 1;
+}
+
+
+byte_t* encode_seq_on_2_bits(const char* seq, int32_t length)
+{
+	byte_t*  seq_b;
+	uint8_t  modulo;
+	int32_t  length_b;
+	int32_t  i;
+
+	length_b = ceil((double) length / (double) 4.0);
+
+	seq_b = (byte_t*) calloc(length_b, sizeof(byte_t));
+	if (seq_b == NULL)
+	{
+		obi_set_errno(OBI_MALLOC_ERROR);
+		obidebug(1, "\nError allocating memory for an encoded DNA sequence");
+		return NULL;
+	}
+
+	for (i=0; i<length; i++)
+	{
+		// Shift of 2 to make place for new nucleotide
+		seq_b[i/4] <<= 2;
+
+		// Add new nucleotide
+		switch (seq[i])
+		{
+		case 'a':
+		case 'A':
+			seq_b[i/4] |= NUC_A_2b;
+			break;
+		case 'c':
+		case 'C':
+			seq_b[i/4] |= NUC_C_2b;
+			break;
+		case 'g':
+		case 'G':
+			seq_b[i/4] |= NUC_G_2b;
+			break;
+		case 't':
+		case 'T':
+			seq_b[i/4] |= NUC_T_2b;
+			break;
+		default:
+			obi_set_errno(OBI_ENCODE_ERROR);
+			obidebug(1, "\nInvalid nucleotide base when encoding (not [atgcATGC])");
+			return NULL;
+		}
+	}
+
+	// Final shift for the last byte if needed
+	modulo = (length % 4);
+	if (modulo)
+		seq_b[(i-1)/4] <<= (2*(4 - modulo));
+
+	return seq_b;
+}
+
+
+char* decode_seq_on_2_bits(byte_t* seq_b, int32_t length_seq)
+{
+	char*   seq;
+	int32_t i;
+	uint8_t shift;
+	uint8_t mask;
+	uint8_t nuc;
+
+	seq = (char*) malloc((length_seq+1) * sizeof(char));
+	if (seq == NULL)
+	{
+		obi_set_errno(OBI_MALLOC_ERROR);
+		obidebug(1, "\nError allocating memory for a decoded DNA sequence");
+		return NULL;
+	}
+
+	for (i=0; i<length_seq; i++)
+	{
+		shift = 6 - 2*(i % 4);
+		mask = NUC_MASK_2B << shift;
+		nuc = (seq_b[i/4] & mask) >> shift;
+
+		switch (nuc)
+		{
+		case NUC_A_2b:
+			seq[i] = 'a';
+			break;
+		case NUC_C_2b:
+			seq[i] = 'c';
+			break;
+		case NUC_G_2b:
+			seq[i] = 'g';
+			break;
+		case NUC_T_2b:
+			seq[i] = 't';
+			break;
+		default:
+			obi_set_errno(OBI_DECODE_ERROR);
+			obidebug(1, "\nInvalid nucleotide base when decoding");
+			return NULL;
+		}
+	}
+
+	seq[length_seq] = '\0';
+
+	return seq;
+}
+
+
+byte_t* encode_seq_on_4_bits(const char* seq, int32_t length)
+{
+	byte_t*  seq_b;
+	uint8_t  modulo;
+	int32_t  length_b;
+	int32_t  i;
+
+	length_b = ceil((double) length / (double) 2.0);
+
+	seq_b = (byte_t*) calloc(length_b, sizeof(byte_t));
+	if (seq_b == NULL)
+	{
+		obi_set_errno(OBI_MALLOC_ERROR);
+		obidebug(1, "\nError allocating memory for an encoded DNA sequence");
+		return NULL;
+	}
+
+	for (i=0; i<length; i++)
+	{
+		// Shift of 4 to make place for new nucleotide
+		seq_b[i/2] <<= 4;
+
+		// Add new nucleotide
+		switch (seq[i])
+		{
+		case 'a':
+		case 'A':
+			seq_b[i/2] |= NUC_A_4b;
+			break;
+		case 'c':
+		case 'C':
+			seq_b[i/2] |= NUC_C_4b;
+			break;
+		case 'g':
+		case 'G':
+			seq_b[i/2] |= NUC_G_4b;
+			break;
+		case 't':
+		case 'T':
+			seq_b[i/2] |= NUC_T_4b;
+			break;
+		case 'r':
+		case 'R':
+			seq_b[i/2] |= NUC_R_4b;
+			break;
+		case 'y':
+		case 'Y':
+			seq_b[i/2] |= NUC_Y_4b;
+			break;
+		case 's':
+		case 'S':
+			seq_b[i/2] |= NUC_S_4b;
+			break;
+		case 'w':
+		case 'W':
+			seq_b[i/2] |= NUC_W_4b;
+			break;
+		case 'k':
+		case 'K':
+			seq_b[i/2] |= NUC_K_4b;
+			break;
+		case 'm':
+		case 'M':
+			seq_b[i/2] |= NUC_M_4b;
+			break;
+		case 'b':
+		case 'B':
+			seq_b[i/2] |= NUC_B_4b;
+			break;
+		case 'd':
+		case 'D':
+			seq_b[i/2] |= NUC_D_4b;
+			break;
+		case 'h':
+		case 'H':
+			seq_b[i/2] |= NUC_H_4b;
+			break;
+		case 'v':
+		case 'V':
+			seq_b[i/2] |= NUC_V_4b;
+			break;
+		case 'n':
+		case 'N':
+			seq_b[i/2] |= NUC_N_4b;
+			break;
+		default:
+			obi_set_errno(OBI_ENCODE_ERROR);
+			obidebug(1, "\nInvalid nucleotide base when encoding (not IUPAC)");
+			return NULL;
+		}
+	}
+
+	// Final shift for the last byte if needed
+	modulo = (length % 2);
+	if (modulo)
+		seq_b[(i-1)/2] <<= (4*modulo);
+
+	return seq_b;
+}
+
+
+char* decode_seq_on_4_bits(byte_t* seq_b, int32_t length_seq)
+{
+	char*   seq;
+	int32_t i;
+	uint8_t shift;
+	uint8_t mask;
+	uint8_t nuc;
+
+	seq = (char*) malloc((length_seq+1) * sizeof(char));
+	if (seq == NULL)
+	{
+		obi_set_errno(OBI_MALLOC_ERROR);
+		obidebug(1, "\nError allocating memory for a decoded DNA sequence");
+		return NULL;
+	}
+
+	for (i=0; i<length_seq; i++)
+	{
+		shift = 4 - 4*(i % 2);
+		mask = NUC_MASK_4B << shift;
+		nuc = (seq_b[i/2] & mask) >> shift;
+
+		switch (nuc)
+		{
+		case NUC_A_4b:
+			seq[i] = 'a';
+			break;
+		case NUC_C_4b:
+			seq[i] = 'c';
+			break;
+		case NUC_G_4b:
+			seq[i] = 'g';
+			break;
+		case NUC_T_4b:
+			seq[i] = 't';
+			break;
+		case NUC_R_4b:
+			seq[i] = 'r';
+			break;
+		case NUC_Y_4b:
+			seq[i] = 'y';
+			break;
+		case NUC_S_4b:
+			seq[i] = 's';
+			break;
+		case NUC_W_4b:
+			seq[i] = 'w';
+			break;
+		case NUC_K_4b:
+			seq[i] = 'k';
+			break;
+		case NUC_M_4b:
+			seq[i] = 'm';
+			break;
+		case NUC_B_4b:
+			seq[i] = 'b';
+			break;
+		case NUC_D_4b:
+			seq[i] = 'd';
+			break;
+		case NUC_H_4b:
+			seq[i] = 'h';
+			break;
+		case NUC_V_4b:
+			seq[i] = 'v';
+			break;
+		case NUC_N_4b:
+			seq[i] = 'n';
+			break;
+		default:
+			obi_set_errno(OBI_DECODE_ERROR);
+			obidebug(1, "\nInvalid nucleotide base when decoding");
+			return NULL;
+		}
+	}
+
+	seq[length_seq] = '\0';
+
+	return seq;
+}
+
+
+///////////////////// FOR DEBUGGING ///////////////////////////
+//NOTE: The first byte is printed the first (at the left-most).
+
+void print_bits(void* ptr, int32_t size)
+{
+	uint8_t* b = (uint8_t*) ptr;
+	uint8_t byte;
+    int32_t i, j;
+
+    fprintf(stderr, "\n");
+    for (i=0;i<size;i++)
+    {
+        for (j=7;j>=0;j--)
+        {
+            byte = b[i] & (1<<j);
+            byte >>= j;
+            fprintf(stderr, "%u", byte);
+        }
+        fprintf(stderr, " ");
+    }
+    fprintf(stderr, "\n");
+}

src/encode.h

@@ -0,0 +1,190 @@
+/****************************************************************************
+ * Encoding header file	                                                    *
+ ****************************************************************************/
+
+/**
+ * @file encode.h
+ * @author Celine Mercier
+ * @date November 18th 2015
+ * @brief Header file for encoding DNA sequences.
+ */
+
+
+#ifndef ENCODE_H_
+#define ENCODE_H_
+
+
+#include <stdlib.h>
+#include <stdio.h>
+#include <stdint.h>
+#include <stdbool.h>
+
+#include "obitypes.h"
+
+
+#define NUC_MASK_2B 0x3   	/**< Binary: 11 to use when decoding 2 bits sequences
+ 	 	 	 	 	 	 	 */
+#define NUC_MASK_4B 0xF  	/**< Binary: 1111 to use when decoding 4 bits sequences
+ 	 	 	 	 	 	 	 */
+
+
+/**
+ * @brief enum for the 2-bits codes for each of the 4 nucleotides.
+ */
+enum
+{
+    NUC_A_2b = 0x0,   /* binary: 00 */
+    NUC_C_2b = 0x1,   /* binary: 01 */
+    NUC_G_2b = 0x2,   /* binary: 10 */
+    NUC_T_2b = 0x3,   /* binary: 11 */
+};
+
+
+/**
+ * @brief enum for the 4-bits codes for each of the 15 IUPAC nucleotides.
+ */
+enum
+{
+    NUC_A_4b = 0x1,   /* binary: 0001 */
+    NUC_C_4b = 0x2,   /* binary: 0010 */
+    NUC_G_4b = 0x3,   /* binary: 0011 */
+    NUC_T_4b = 0x4,   /* binary: 0100 */
+	NUC_R_4b = 0x5,   /* binary: 0101 */
+	NUC_Y_4b = 0x6,   /* binary: 0110 */
+	NUC_S_4b = 0x7,   /* binary: 0111 */
+	NUC_W_4b = 0x8,   /* binary: 1000 */
+	NUC_K_4b = 0x9,   /* binary: 1001 */
+	NUC_M_4b = 0xA,   /* binary: 1010 */
+	NUC_B_4b = 0xB,   /* binary: 1011 */
+	NUC_D_4b = 0xC,   /* binary: 1100 */
+	NUC_H_4b = 0xD,   /* binary: 1101 */
+	NUC_V_4b = 0xE,   /* binary: 1110 */
+	NUC_N_4b = 0xF,   /* binary: 1111 */
+};
+
+
+/**
+ * @brief Checks if there are only 'atgcATGC' characters in a
+ *        character string.
+ *
+ * @param seq The sequence to check.
+ *
+ * @returns A boolean value indicating if there are only
+ *          'atgcATGC' characters in a character string.
+ *
+ * @since November 2015
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+bool only_ATGC(const char* seq);
+
+
+/**
+ * @brief Encodes a DNA sequence with each nucleotide coded on 2 bits.
+ *
+ *    A or a : 00
+ *    C or c : 01
+ *    T or t : 10
+ *    G or g : 11
+ *
+ * @warning The DNA sequence must contain only 'atgcATGC' characters.
+ *
+ * @param seq The sequence to encode.
+ * @param length The length of the sequence to encode.
+ *
+ * @returns The encoded sequence.
+ *
+ * @since November 2015
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+byte_t* encode_seq_on_2_bits(const char* seq, int32_t length);
+
+
+/**
+ * @brief Decodes a DNA sequence that is coded with each nucleotide on 2 bits.
+ *
+ *    00 -> a
+ *    01 -> c
+ *    10 -> t
+ *    11 -> g
+ *
+ * @param seq_b The sequence to decode.
+ * @param length_seq The initial length of the sequence before it was encoded.
+ *
+ * @returns The decoded sequence ended with '\0'.
+ *
+ * @since November 2015
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+char* decode_seq_on_2_bits(byte_t* seq_b, int32_t length_seq);
+
+
+/**
+ * @brief Encodes a DNA sequence with each nucleotide coded on 4 bits.
+ *
+ *		A or a : 0001
+ *      C or c : 0010
+ *      G or g : 0011
+ *      T or t : 0100
+ *      R or r : 0101
+ *      Y or y : 0110
+ *      S or s : 0111
+ *      W or w : 1000
+ *      K or k : 1001
+ *      M or m : 1010
+ *      B or b : 1011
+ *      D or d : 1100
+ *      H or h : 1101
+ *      V or v : 1110
+ *      N or n : 1111
+ *
+ * @warning The DNA sequence must contain only IUPAC characters.
+ *
+ * @param seq The sequence to encode.
+ * @param length The length of the sequence to encode.
+ *
+ * @returns The encoded sequence.
+ *
+ * @since November 2015
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+byte_t* encode_seq_on_4_bits(const char* seq, int32_t length);
+
+
+/**
+ * @brief Decodes a DNA sequence that is coded with each nucleotide on 4 bits.
+ *
+ *		A or a : 0001
+ *      C or c : 0010
+ *      G or g : 0011
+ *      T or t : 0100
+ *      R or r : 0101
+ *      Y or y : 0110
+ *      S or s : 0111
+ *      W or w : 1000
+ *      K or k : 1001
+ *      M or m : 1010
+ *      B or b : 1011
+ *      D or d : 1100
+ *      H or h : 1101
+ *      V or v : 1110
+ *      N or n : 1111
+ *
+ * @param seq_b The sequence to decode.
+ * @param length_seq The initial length of the sequence before it was encoded.
+ *
+ * @returns The decoded sequence ended with '\0'.
+ *
+ * @since November 2015
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+char* decode_seq_on_4_bits(byte_t* seq_b, int32_t length_seq);
+
+
+////////// FOR DEBUGGING ///////////
+
+// little endian
+void print_bits(void* ptr, int32_t length);
+
+
+#endif /* ENCODE_H_ */
+

src/murmurhash2.c

@@ -0,0 +1,64 @@
+//-----------------------------------------------------------------------------
+// MurmurHash2, by Austin Appleby
+
+// Note - This code makes a few assumptions about how your machine behaves -
+
+// 1. We can read a 4-byte value from any address without crashing
+// 2. sizeof(int) == 4
+
+// And it has a few limitations -
+
+// 1. It will not work incrementally.
+// 2. It will not produce the same results on little-endian and big-endian
+//    machines.	->TODO
+
+unsigned int murmurhash2(const void * key, int len, const unsigned int seed)
+{
+	// 'm' and 'r' are mixing constants generated offline.
+	// They're not really 'magic', they just happen to work well.
+
+	const unsigned int m = 0x5bd1e995;
+	const int r = 24;
+
+	// Initialize the hash to a 'random' value
+
+	unsigned int h = seed ^ len;
+
+	// Mix 4 bytes at a time into the hash
+
+	const unsigned char * data = (const unsigned char *)key;
+
+	while(len >= 4)
+	{
+		unsigned int k = *(unsigned int *)data;
+
+		k *= m;
+		k ^= k >> r;
+		k *= m;
+
+		h *= m;
+		h ^= k;
+
+		data += 4;
+		len -= 4;
+	}
+
+	// Handle the last few bytes of the input array
+
+	switch(len)
+	{
+	case 3: h ^= data[2] << 16;
+	case 2: h ^= data[1] << 8;
+	case 1: h ^= data[0];
+	        h *= m;
+	};
+
+	// Do a few final mixes of the hash to ensure the last few
+	// bytes are well-incorporated.
+
+	h ^= h >> 13;
+	h *= m;
+	h ^= h >> 15;
+
+	return h;
+}

src/murmurhash2.h

@@ -0,0 +1,11 @@
+
+#ifndef _BLOOM_MURMURHASH2
+#define _BLOOM_MURMURHASH2
+
+
+/**
+ * @brief Generates and returns a hash code from a byte array.
+ */
+unsigned int murmurhash2(const void * key, int len, const unsigned int seed);
+
+#endif

src/obi_align.c

@@ -0,0 +1,109 @@
+/****************************************************************************
+ * Sequence alignment functions				                                *
+ ****************************************************************************/
+
+/**
+ * @file obi_align.c
+ * @author Celine Mercier
+ * @date May 4th 2016
+ * @brief Functions handling sequence alignments.
+ */
+
+
+#include <stdlib.h>
+#include <stdio.h>
+#include <stdbool.h>
+
+#include "obidebug.h"
+#include "obierrno.h"
+#include "obitypes.h"
+#include "obiview.h"
+#include "sse_banded_LCS_alignment.h"
+
+
+#define DEBUG_LEVEL 0	// TODO has to be defined somewhere else (cython compil flag?)
+
+
+// TODO
+// use openMP pragmas : garder scores en memoire et ecrire a la fin? (normalement c bon avec index)
+// tout ecrire en stdint?
+// check NUC_SEQS and score type (int or float if normalize)
+// what's with multiple sequence/line columns?
+// make function that put blobs in int16
+
+
+int obi_align_one_column(Obiview_p seq_view, OBIDMS_column_p seq_column, Obiview_p score_view, OBIDMS_column_p score_column, double threshold, bool normalize, int reference, bool similarity_mode)
+{
+	index_t i, j, k;
+	index_t seq_count;
+	char* seq1;
+	char* seq2;
+	double score;
+
+	k = 0;
+
+	if ((seq_column->header)->returned_data_type != OBI_SEQ)
+	{
+		obi_set_errno(OBI_ALIGN_ERROR);
+		obidebug(1, "\nTrying to align a column of a different type than OBI_SEQ");
+		return -1;
+	}
+
+	if ((normalize && ((score_column->header)->returned_data_type != OBI_FLOAT)) ||
+			(!normalize && ((score_column->header)->returned_data_type != OBI_INT)))
+	{
+		obi_set_errno(OBI_ALIGN_ERROR);
+		obidebug(1, "\nTrying to store alignment scores in a column of an inappropriate type");
+		return -1;
+	}
+
+	seq_count = (seq_column->header)->lines_used;
+
+	for (i=0; i < (seq_count - 1); i++)
+	{
+		for (j=i+1; j < seq_count; j++)
+		{
+			//fprintf(stderr, "\ni=%lld, j=%lld, k=%lld", i, j, k);
+
+			seq1 = obi_column_get_obiseq_with_elt_idx_in_view(seq_view, seq_column, i, 0);
+			seq2 = obi_column_get_obiseq_with_elt_idx_in_view(seq_view, seq_column, j, 0);
+
+			if ((seq1 == NULL) || (seq2 == NULL))
+			{
+				obidebug(1, "\nError retrieving sequences to align");
+				return -1;
+			}
+
+			// TODO filter
+			score = generic_sse_banded_lcs_align(seq1, seq2, threshold, normalize, reference, similarity_mode);
+
+			if (normalize)
+			{
+				if (obi_column_set_obifloat_with_elt_idx_in_view(score_view, score_column, k, 0, (obifloat_t) score) < 0)
+				{
+					obidebug(1, "\nError writing alignment score in a column");
+					return -1;
+				}
+			}
+			else
+			{
+				if (obi_column_set_obiint_with_elt_idx_in_view(score_view, score_column, k, 0, (obiint_t) score) < 0)
+				{
+					obidebug(1, "\nError writing alignment score in a column");
+					return -1;
+				}
+			}
+
+			k++;
+		}
+	}
+
+	return 0;
+}
+
+
+int obi_align_two_columns(OBIDMS_column_p seq_column_1, OBIDMS_column_p seq_column_2, OBIDMS_column_p score_column, double threshold, bool normalize, int reference, bool similarity_mode)
+{
+	// TODO
+	return 0;
+}