Simplify the code

Conflicts:
	python/obitools3/obidms/_obidmscolumn_seq.pyx

LICENSE

@@ -0,0 +1,518 @@
+
+  CeCILL FREE SOFTWARE LICENSE AGREEMENT
+
+Version 2.1 dated 2013-06-21
+
+
+    Notice
+
+This Agreement is a Free Software license agreement that is the result
+of discussions between its authors in order to ensure compliance with
+the two main principles guiding its drafting:
+
+  * firstly, compliance with the principles governing the distribution
+    of Free Software: access to source code, broad rights granted to users,
+  * secondly, the election of a governing law, French law, with which it
+    is conformant, both as regards the law of torts and intellectual
+    property law, and the protection that it offers to both authors and
+    holders of the economic rights over software.
+
+The authors of the CeCILL (for Ce[a] C[nrs] I[nria] L[ogiciel] L[ibre]) 
+license are: 
+
+Commissariat à l'énergie atomique et aux énergies alternatives - CEA, a
+public scientific, technical and industrial research establishment,
+having its principal place of business at 25 rue Leblanc, immeuble Le
+Ponant D, 75015 Paris, France.
+
+Centre National de la Recherche Scientifique - CNRS, a public scientific
+and technological establishment, having its principal place of business
+at 3 rue Michel-Ange, 75794 Paris cedex 16, France.
+
+Institut National de Recherche en Informatique et en Automatique -
+Inria, a public scientific and technological establishment, having its
+principal place of business at Domaine de Voluceau, Rocquencourt, BP
+105, 78153 Le Chesnay cedex, France.
+
+
+    Preamble
+
+The purpose of this Free Software license agreement is to grant users
+the right to modify and redistribute the software governed by this
+license within the framework of an open source distribution model.
+
+The exercising of this right is conditional upon certain obligations for
+users so as to preserve this status for all subsequent redistributions.
+
+In consideration of access to the source code and the rights to copy,
+modify and redistribute granted by the license, users are provided only
+with a limited warranty and the software's author, the holder of the
+economic rights, and the successive licensors only have limited liability.
+
+In this respect, the risks associated with loading, using, modifying
+and/or developing or reproducing the software by the user are brought to
+the user's attention, given its Free Software status, which may make it
+complicated to use, with the result that its use is reserved for
+developers and experienced professionals having in-depth computer
+knowledge. Users are therefore encouraged to load and test the
+suitability of the software as regards their requirements in conditions
+enabling the security of their systems and/or data to be ensured and,
+more generally, to use and operate it in the same conditions of
+security. This Agreement may be freely reproduced and published,
+provided it is not altered, and that no provisions are either added or
+removed herefrom.
+
+This Agreement may apply to any or all software for which the holder of
+the economic rights decides to submit the use thereof to its provisions.
+
+Frequently asked questions can be found on the official website of the
+CeCILL licenses family (http://www.cecill.info/index.en.html) for any 
+necessary clarification.
+
+
+    Article 1 - DEFINITIONS
+
+For the purpose of this Agreement, when the following expressions
+commence with a capital letter, they shall have the following meaning:
+
+Agreement: means this license agreement, and its possible subsequent
+versions and annexes.
+
+Software: means the software in its Object Code and/or Source Code form
+and, where applicable, its documentation, "as is" when the Licensee
+accepts the Agreement.
+
+Initial Software: means the Software in its Source Code and possibly its
+Object Code form and, where applicable, its documentation, "as is" when
+it is first distributed under the terms and conditions of the Agreement.
+
+Modified Software: means the Software modified by at least one
+Contribution.
+
+Source Code: means all the Software's instructions and program lines to
+which access is required so as to modify the Software.
+
+Object Code: means the binary files originating from the compilation of
+the Source Code.
+
+Holder: means the holder(s) of the economic rights over the Initial
+Software.
+
+Licensee: means the Software user(s) having accepted the Agreement.
+
+Contributor: means a Licensee having made at least one Contribution.
+
+Licensor: means the Holder, or any other individual or legal entity, who
+distributes the Software under the Agreement.
+
+Contribution: means any or all modifications, corrections, translations,
+adaptations and/or new functions integrated into the Software by any or
+all Contributors, as well as any or all Internal Modules.
+
+Module: means a set of sources files including their documentation that
+enables supplementary functions or services in addition to those offered
+by the Software.
+
+External Module: means any or all Modules, not derived from the
+Software, so that this Module and the Software run in separate address
+spaces, with one calling the other when they are run.
+
+Internal Module: means any or all Module, connected to the Software so
+that they both execute in the same address space.
+
+GNU GPL: means the GNU General Public License version 2 or any
+subsequent version, as published by the Free Software Foundation Inc.
+
+GNU Affero GPL: means the GNU Affero General Public License version 3 or
+any subsequent version, as published by the Free Software Foundation Inc.
+
+EUPL: means the European Union Public License version 1.1 or any
+subsequent version, as published by the European Commission.
+
+Parties: mean both the Licensee and the Licensor.
+
+These expressions may be used both in singular and plural form.
+
+
+    Article 2 - PURPOSE
+
+The purpose of the Agreement is the grant by the Licensor to the
+Licensee of a non-exclusive, transferable and worldwide license for the
+Software as set forth in Article 5 <#scope> hereinafter for the whole
+term of the protection granted by the rights over said Software.
+
+
+    Article 3 - ACCEPTANCE
+
+3.1 The Licensee shall be deemed as having accepted the terms and
+conditions of this Agreement upon the occurrence of the first of the
+following events:
+
+  * (i) loading the Software by any or all means, notably, by
+    downloading from a remote server, or by loading from a physical medium;
+  * (ii) the first time the Licensee exercises any of the rights granted
+    hereunder.
+
+3.2 One copy of the Agreement, containing a notice relating to the
+characteristics of the Software, to the limited warranty, and to the
+fact that its use is restricted to experienced users has been provided
+to the Licensee prior to its acceptance as set forth in Article 3.1
+<#accepting> hereinabove, and the Licensee hereby acknowledges that it
+has read and understood it.
+
+
+    Article 4 - EFFECTIVE DATE AND TERM
+
+
+      4.1 EFFECTIVE DATE
+
+The Agreement shall become effective on the date when it is accepted by
+the Licensee as set forth in Article 3.1 <#accepting>.
+
+
+      4.2 TERM
+
+The Agreement shall remain in force for the entire legal term of
+protection of the economic rights over the Software.
+
+
+    Article 5 - SCOPE OF RIGHTS GRANTED
+
+The Licensor hereby grants to the Licensee, who accepts, the following
+rights over the Software for any or all use, and for the term of the
+Agreement, on the basis of the terms and conditions set forth hereinafter.
+
+Besides, if the Licensor owns or comes to own one or more patents
+protecting all or part of the functions of the Software or of its
+components, the Licensor undertakes not to enforce the rights granted by
+these patents against successive Licensees using, exploiting or
+modifying the Software. If these patents are transferred, the Licensor
+undertakes to have the transferees subscribe to the obligations set
+forth in this paragraph.
+
+
+      5.1 RIGHT OF USE
+
+The Licensee is authorized to use the Software, without any limitation
+as to its fields of application, with it being hereinafter specified
+that this comprises:
+
+ 1. permanent or temporary reproduction of all or part of the Software
+    by any or all means and in any or all form.
+
+ 2. loading, displaying, running, or storing the Software on any or all
+    medium.
+
+ 3. entitlement to observe, study or test its operation so as to
+    determine the ideas and principles behind any or all constituent
+    elements of said Software. This shall apply when the Licensee
+    carries out any or all loading, displaying, running, transmission or
+    storage operation as regards the Software, that it is entitled to
+    carry out hereunder.
+
+
+      5.2 ENTITLEMENT TO MAKE CONTRIBUTIONS
+
+The right to make Contributions includes the right to translate, adapt,
+arrange, or make any or all modifications to the Software, and the right
+to reproduce the resulting software.
+
+The Licensee is authorized to make any or all Contributions to the
+Software provided that it includes an explicit notice that it is the
+author of said Contribution and indicates the date of the creation thereof.
+
+
+      5.3 RIGHT OF DISTRIBUTION
+
+In particular, the right of distribution includes the right to publish,
+transmit and communicate the Software to the general public on any or
+all medium, and by any or all means, and the right to market, either in
+consideration of a fee, or free of charge, one or more copies of the
+Software by any means.
+
+The Licensee is further authorized to distribute copies of the modified
+or unmodified Software to third parties according to the terms and
+conditions set forth hereinafter.
+
+
+        5.3.1 DISTRIBUTION OF SOFTWARE WITHOUT MODIFICATION
+
+The Licensee is authorized to distribute true copies of the Software in
+Source Code or Object Code form, provided that said distribution
+complies with all the provisions of the Agreement and is accompanied by:
+
+ 1. a copy of the Agreement,
+
+ 2. a notice relating to the limitation of both the Licensor's warranty
+    and liability as set forth in Articles 8 and 9,
+
+and that, in the event that only the Object Code of the Software is
+redistributed, the Licensee allows effective access to the full Source
+Code of the Software for a period of at least three years from the
+distribution of the Software, it being understood that the additional
+acquisition cost of the Source Code shall not exceed the cost of the
+data transfer.
+
+
+        5.3.2 DISTRIBUTION OF MODIFIED SOFTWARE
+
+When the Licensee makes a Contribution to the Software, the terms and
+conditions for the distribution of the resulting Modified Software
+become subject to all the provisions of this Agreement.
+
+The Licensee is authorized to distribute the Modified Software, in
+source code or object code form, provided that said distribution
+complies with all the provisions of the Agreement and is accompanied by:
+
+ 1. a copy of the Agreement,
+
+ 2. a notice relating to the limitation of both the Licensor's warranty
+    and liability as set forth in Articles 8 and 9,
+
+and, in the event that only the object code of the Modified Software is
+redistributed,
+
+ 3. a note stating the conditions of effective access to the full source
+    code of the Modified Software for a period of at least three years
+    from the distribution of the Modified Software, it being understood
+    that the additional acquisition cost of the source code shall not
+    exceed the cost of the data transfer.
+
+
+        5.3.3 DISTRIBUTION OF EXTERNAL MODULES
+
+When the Licensee has developed an External Module, the terms and
+conditions of this Agreement do not apply to said External Module, that
+may be distributed under a separate license agreement.
+
+
+        5.3.4 COMPATIBILITY WITH OTHER LICENSES
+
+The Licensee can include a code that is subject to the provisions of one
+of the versions of the GNU GPL, GNU Affero GPL and/or EUPL in the
+Modified or unmodified Software, and distribute that entire code under
+the terms of the same version of the GNU GPL, GNU Affero GPL and/or EUPL.
+
+The Licensee can include the Modified or unmodified Software in a code
+that is subject to the provisions of one of the versions of the GNU GPL,
+GNU Affero GPL and/or EUPL and distribute that entire code under the
+terms of the same version of the GNU GPL, GNU Affero GPL and/or EUPL.
+
+
+    Article 6 - INTELLECTUAL PROPERTY
+
+
+      6.1 OVER THE INITIAL SOFTWARE
+
+The Holder owns the economic rights over the Initial Software. Any or
+all use of the Initial Software is subject to compliance with the terms
+and conditions under which the Holder has elected to distribute its work
+and no one shall be entitled to modify the terms and conditions for the
+distribution of said Initial Software.
+
+The Holder undertakes that the Initial Software will remain ruled at
+least by this Agreement, for the duration set forth in Article 4.2 <#term>.
+
+
+      6.2 OVER THE CONTRIBUTIONS
+
+The Licensee who develops a Contribution is the owner of the
+intellectual property rights over this Contribution as defined by
+applicable law.
+
+
+      6.3 OVER THE EXTERNAL MODULES
+
+The Licensee who develops an External Module is the owner of the
+intellectual property rights over this External Module as defined by
+applicable law and is free to choose the type of agreement that shall
+govern its distribution.
+
+
+      6.4 JOINT PROVISIONS
+
+The Licensee expressly undertakes:
+
+ 1. not to remove, or modify, in any manner, the intellectual property
+    notices attached to the Software;
+
+ 2. to reproduce said notices, in an identical manner, in the copies of
+    the Software modified or not.
+
+The Licensee undertakes not to directly or indirectly infringe the
+intellectual property rights on the Software of the Holder and/or
+Contributors, and to take, where applicable, vis-à-vis its staff, any
+and all measures required to ensure respect of said intellectual
+property rights of the Holder and/or Contributors.
+
+
+    Article 7 - RELATED SERVICES
+
+7.1 Under no circumstances shall the Agreement oblige the Licensor to
+provide technical assistance or maintenance services for the Software.
+
+However, the Licensor is entitled to offer this type of services. The
+terms and conditions of such technical assistance, and/or such
+maintenance, shall be set forth in a separate instrument. Only the
+Licensor offering said maintenance and/or technical assistance services
+shall incur liability therefor.
+
+7.2 Similarly, any Licensor is entitled to offer to its licensees, under
+its sole responsibility, a warranty, that shall only be binding upon
+itself, for the redistribution of the Software and/or the Modified
+Software, under terms and conditions that it is free to decide. Said
+warranty, and the financial terms and conditions of its application,
+shall be subject of a separate instrument executed between the Licensor
+and the Licensee.
+
+
+    Article 8 - LIABILITY
+
+8.1 Subject to the provisions of Article 8.2, the Licensee shall be
+entitled to claim compensation for any direct loss it may have suffered
+from the Software as a result of a fault on the part of the relevant
+Licensor, subject to providing evidence thereof.
+
+8.2 The Licensor's liability is limited to the commitments made under
+this Agreement and shall not be incurred as a result of in particular:
+(i) loss due the Licensee's total or partial failure to fulfill its
+obligations, (ii) direct or consequential loss that is suffered by the
+Licensee due to the use or performance of the Software, and (iii) more
+generally, any consequential loss. In particular the Parties expressly
+agree that any or all pecuniary or business loss (i.e. loss of data,
+loss of profits, operating loss, loss of customers or orders,
+opportunity cost, any disturbance to business activities) or any or all
+legal proceedings instituted against the Licensee by a third party,
+shall constitute consequential loss and shall not provide entitlement to
+any or all compensation from the Licensor.
+
+
+    Article 9 - WARRANTY
+
+9.1 The Licensee acknowledges that the scientific and technical
+state-of-the-art when the Software was distributed did not enable all
+possible uses to be tested and verified, nor for the presence of
+possible defects to be detected. In this respect, the Licensee's
+attention has been drawn to the risks associated with loading, using,
+modifying and/or developing and reproducing the Software which are
+reserved for experienced users.
+
+The Licensee shall be responsible for verifying, by any or all means,
+the suitability of the product for its requirements, its good working
+order, and for ensuring that it shall not cause damage to either persons
+or properties.
+
+9.2 The Licensor hereby represents, in good faith, that it is entitled
+to grant all the rights over the Software (including in particular the
+rights set forth in Article 5 <#scope>).
+
+9.3 The Licensee acknowledges that the Software is supplied "as is" by
+the Licensor without any other express or tacit warranty, other than
+that provided for in Article 9.2 <#good-faith> and, in particular,
+without any warranty as to its commercial value, its secured, safe,
+innovative or relevant nature.
+
+Specifically, the Licensor does not warrant that the Software is free
+from any error, that it will operate without interruption, that it will
+be compatible with the Licensee's own equipment and software
+configuration, nor that it will meet the Licensee's requirements.
+
+9.4 The Licensor does not either expressly or tacitly warrant that the
+Software does not infringe any third party intellectual property right
+relating to a patent, software or any other property right. Therefore,
+the Licensor disclaims any and all liability towards the Licensee
+arising out of any or all proceedings for infringement that may be
+instituted in respect of the use, modification and redistribution of the
+Software. Nevertheless, should such proceedings be instituted against
+the Licensee, the Licensor shall provide it with technical and legal
+expertise for its defense. Such technical and legal expertise shall be
+decided on a case-by-case basis between the relevant Licensor and the
+Licensee pursuant to a memorandum of understanding. The Licensor
+disclaims any and all liability as regards the Licensee's use of the
+name of the Software. No warranty is given as regards the existence of
+prior rights over the name of the Software or as regards the existence
+of a trademark.
+
+
+    Article 10 - TERMINATION
+
+10.1 In the event of a breach by the Licensee of its obligations
+hereunder, the Licensor may automatically terminate this Agreement
+thirty (30) days after notice has been sent to the Licensee and has
+remained ineffective.
+
+10.2 A Licensee whose Agreement is terminated shall no longer be
+authorized to use, modify or distribute the Software. However, any
+licenses that it may have granted prior to termination of the Agreement
+shall remain valid subject to their having been granted in compliance
+with the terms and conditions hereof.
+
+
+    Article 11 - MISCELLANEOUS
+
+
+      11.1 EXCUSABLE EVENTS
+
+Neither Party shall be liable for any or all delay, or failure to
+perform the Agreement, that may be attributable to an event of force
+majeure, an act of God or an outside cause, such as defective
+functioning or interruptions of the electricity or telecommunications
+networks, network paralysis following a virus attack, intervention by
+government authorities, natural disasters, water damage, earthquakes,
+fire, explosions, strikes and labor unrest, war, etc.
+
+11.2 Any failure by either Party, on one or more occasions, to invoke
+one or more of the provisions hereof, shall under no circumstances be
+interpreted as being a waiver by the interested Party of its right to
+invoke said provision(s) subsequently.
+
+11.3 The Agreement cancels and replaces any or all previous agreements,
+whether written or oral, between the Parties and having the same
+purpose, and constitutes the entirety of the agreement between said
+Parties concerning said purpose. No supplement or modification to the
+terms and conditions hereof shall be effective as between the Parties
+unless it is made in writing and signed by their duly authorized
+representatives.
+
+11.4 In the event that one or more of the provisions hereof were to
+conflict with a current or future applicable act or legislative text,
+said act or legislative text shall prevail, and the Parties shall make
+the necessary amendments so as to comply with said act or legislative
+text. All other provisions shall remain effective. Similarly, invalidity
+of a provision of the Agreement, for any reason whatsoever, shall not
+cause the Agreement as a whole to be invalid.
+
+
+      11.5 LANGUAGE
+
+The Agreement is drafted in both French and English and both versions
+are deemed authentic.
+
+
+    Article 12 - NEW VERSIONS OF THE AGREEMENT
+
+12.1 Any person is authorized to duplicate and distribute copies of this
+Agreement.
+
+12.2 So as to ensure coherence, the wording of this Agreement is
+protected and may only be modified by the authors of the License, who
+reserve the right to periodically publish updates or new versions of the
+Agreement, each with a separate number. These subsequent versions may
+address new issues encountered by Free Software.
+
+12.3 Any Software distributed under a given version of the Agreement may
+only be subsequently distributed under the same version of the Agreement
+or a subsequent version, subject to the provisions of Article 5.3.4
+<#compatibility>.
+
+
+    Article 13 - GOVERNING LAW AND JURISDICTION
+
+13.1 The Agreement is governed by French law. The Parties agree to
+endeavor to seek an amicable solution to any disagreements or disputes
+that may arise during the performance of the Agreement.
+
+13.2 Failing an amicable solution within two (2) months as from their
+occurrence, and unless emergency proceedings are necessary, the
+disagreements or disputes shall be referred to the Paris Courts having
+jurisdiction, by the more diligent Party.

README.md

@@ -0,0 +1,40 @@
+The `OBITools3`: A package for the management of analyses and data in DNA metabarcoding   
+---------------------------------------------
+
+
+
+DNA metabarcoding offers new perspectives for biodiversity research [1]. This approach of ecosystem studies relies heavily on the use of Next-Generation Sequencing (NGS), and consequently requires the ability to to treat large volumes of data. The `OBITools` package  satisfies this requirement thanks to a set of programs specifically designed for analyzing NGS data in a DNA metabarcoding context [2] - <http://metabarcoding.org/obitools>. Their capacity to filter and edit sequences while taking into account taxonomic annotation helps to setup tailored-made analysis pipelines for a broad range of DNA metabarcoding applications, including biodiversity surveys or diet analyses.   
+
+
+
+**The `OBITools3`.** This new version of the `OBITools` looks to significantly improve the storage efficiency and the data processing speed. To this end, the `OBITools3` rely on an ad hoc database system, inside which all the data that a DNA metabarcoding experiment must consider is stored: the sequences, the metadata (describing for instance the samples), the database containing the reference sequences used for the taxonomic annotation, as well as the taxonomic databases. Besides the gain in efficiency, this new structure allows an easier access to all the data associated with an experiment.   
+
+
+
+**Column-oriented storage.** An analysis pipeline corresponds to a succession of commands, each computing one step of the analysis, and where the result of the command *n* is used by the command *n+1*. DNA metabarcoding data can easily be represented in the form of tables, and each command can be regarded as an operation transforming one or several 'input' tables into one or several 'output' tables, which can be used by the next command. Many of the basic operations in a pipeline copy without modification an important part of the input tables to the result tables, and use for their calculations only a small part of the input data. In the original `OBITools`, those tables are kept in the form of annotated sequence files in the FASTA or FASTQ format. This has two consequences: i) keeping the transitional results of the analysis pipeline means using disk space for an important volume of redundant data, ii) The coding and decoding of informations that are not actually used represent an important part of the treatment process. The new database system used by the `OBITools3` (called DMS for Data Management System) relies on column-oriented storage. The columns are immutable and can be assembled in views representing the data tables. This way, the data not modified by a command in an input table can easily be associated to the result table without duplicating any information ; and the data not used at all by a command can be associated with the result table without being read. This strategy results in a gain in disk space efficiency by limiting data redundancy, as well as a gain in execution time by limiting data reading, writing and conversion operations. Finally, as a mean to optimize data access, each column is stored in a binary file directly mapped in memory for reading and writing operations.   
+
+
+
+**Storage optimization.** DNA metabarcoding data is intrinsically very redundant. For example, the same sequence corresponding to a species will be present several thousand times across all samples. In order to limit the disk space used and make comparison operations more efficient, data in the form of character strings is stored in columns using a complex indexing structure, efficient on millions of values, coupling hash functions, Bloom filters and AVL trees. Finally, DNA sequences are compressed by encoding each nucleotide on two or four bits depending on whether the sequences contain only the four nucleotides (A, C, G, T) or use the IUPAC codes.   
+
+
+
+**Saving the data processing history.** The totality of the informations used by the `OBITools3` is stored in immutable data structures in the DMS. If a command has to modify a column used as input to produce its result, a new version of that column is created, leaving the initial version intact. This storage system enables to keep, at minimal cost, the totality of the transitional results produced by the pipeline. The storage of metadata describing all the operations that have produced a view (a result table) in the DMS makes possible the creation of an oriented hypergraph, where each node corresponds to a view and each arrow to an operation. By retracing the dependency relationships in this hypergraph, it is possible to rebuild *a posteriori* the entirety of the process that has produced a result table.   
+
+
+
+**Tools.** The `OBITools3` offer the same tools as the original `OBITools`. Eventually, new versions of `ecoPrimers` (PCR primer design) [3], `ecoPCR` (*in silico* PCR) [4], as well as `Sumatra` (sequence alignment) and `Sumaclust` (sequence alignment and clustering) [5] will be added, taking advantage of the database structure developed for the `OBITools3`.    
+
+
+
+**Implementation and disponibility.** The lower layers managing the DMS as well as all the compute-intensive functions are coded in `C99` for efficiency reasons. A `Cython` (<http://www.cython.org>) object layer allows for a simple but efficient implementation of the `OBITools3` commands in `Python 3.5`. The `OBITools3` are still in development, and the first functional versions are expected for autumn 2016.    
+
+
+
+**References.**
+
+1. Taberlet P, Coissac E, Hajibabaei M, Rieseberg LH: Environmental DNA. Mol Ecol 2012:1789–1793.
+2. Boyer F, Mercier C, Bonin A, Le Bras Y, Taberlet P, Coissac E: OBITools: a Unix-inspired software package for DNA metabarcoding. Mol Ecol Resour 2015:n/a–n/a.
+3. Riaz T, Shehzad W, Viari A, Pompanon F, Taberlet P, Coissac E: ecoPrimers: inference of new DNA barcode markers from whole genome sequence analysis. Nucleic Acids Res 2011, 39:e145.
+4. Ficetola GF, Coissac E, Zundel S, Riaz T, Shehzad W, Bessière J, Taberlet P, Pompanon F: An in silico approach for the evaluation of DNA barcodes. BMC Genomics 2010, 11:434.
+5. Mercier C, Boyer F, Bonin A, Coissac E (2013) SUMATRA and SUMACLUST: fast and exact comparison and clustering of sequences. Available: <http://metabarcoding.org/sumatra> and <http://metabarcoding.org/sumaclust>

distutils.ext/obidistutils/command/build_ext.py

@@ -7,106 +7,108 @@ Created on 13 fevr. 2014
 from distutils import log
 import os
 
-from Cython.Distutils import build_ext  as ori_build_ext  # @UnresolvedImport
-
-from Cython.Compiler import Options as cython_options  # @UnresolvedImport
 
 from distutils.errors import DistutilsSetupError
 
-class build_ext(ori_build_ext):
+try:
+    from Cython.Distutils import build_ext  as ori_build_ext  # @UnresolvedImport
+    from Cython.Compiler import Options as cython_options  # @UnresolvedImport
+    class build_ext(ori_build_ext):
     
         
-    def modifyDocScripts(self):
-        build_dir_file=open("doc/sphinx/build_dir.txt","w")
-        print(self.build_lib,file=build_dir_file)
-        build_dir_file.close()
+        def modifyDocScripts(self):
+            build_dir_file=open("doc/sphinx/build_dir.txt","w")
+            print(self.build_lib,file=build_dir_file)
+            build_dir_file.close()
             
-    def initialize_options(self):
-        ori_build_ext.initialize_options(self)  # @UndefinedVariable
-        self.littlebigman = None
-        self.built_files = None
+        def initialize_options(self):
+            ori_build_ext.initialize_options(self)  # @UndefinedVariable
+            self.littlebigman = None
+            self.built_files = None
     
         
-    def finalize_options(self):
-        ori_build_ext.finalize_options(self)  # @UndefinedVariable
+        def finalize_options(self):
+            ori_build_ext.finalize_options(self)  # @UndefinedVariable
     
-        self.set_undefined_options('littlebigman',
-                                   ('littlebigman',  'littlebigman'))
+            self.set_undefined_options('littlebigman',
+                                       ('littlebigman',  'littlebigman'))
             
-        self.set_undefined_options('build_files',
-                                   ('files',  'built_files'))
+            self.set_undefined_options('build_files',
+                                       ('files',  'built_files'))
     
-        self.cython_c_in_temp = 1
+            self.cython_c_in_temp = 1
                        
-        if self.littlebigman =='-DLITTLE_END':
-            if self.define is None:
-                self.define=[('LITTLE_END',None)]
-            else:
-                self.define.append('LITTLE_END',None)
+            if self.littlebigman =='-DLITTLE_END':
+                if self.define is None:
+                    self.define=[('LITTLE_END',None)]
+                else:
+                    self.define.append('LITTLE_END',None)
             
-    def substitute_sources(self,exe_name,sources):
-        """
-        Substitutes source file name starting by an @ by the actual
-        name of the built file (see --> build_files)
-        """
-        sources = list(sources)
-        for i in range(len(sources)):
-            message = "%s :-> %s" % (exe_name,sources[i])
-            if sources[i][0]=='@':
-                try:
-                    filename = self.built_files[sources[i][1:]]
-                except KeyError:
-                    tmpfilename = os.path.join(self.build_temp,sources[i][1:])
-                    if os.path.isfile       (tmpfilename):
-                        filename = tmpfilename
-                    else:
-                        raise DistutilsSetupError(
-                             'The %s filename declared in the source '
-                             'files of the program %s have not been '
-                             'built by the installation process' % (sources[i],
-                                                                     exe_name))
-                sources[i]=filename
-                log.info("%s changed to %s",message,filename)
-            else:
-                log.info("%s ok",message)
+        def substitute_sources(self,exe_name,sources):
+            """
+            Substitutes source file name starting by an @ by the actual
+            name of the built file (see --> build_files)
+            """
+            sources = list(sources)
+            for i in range(len(sources)):
+                message = "%s :-> %s" % (exe_name,sources[i])
+                if sources[i][0]=='@':
+                    try:
+                        filename = self.built_files[sources[i][1:]]
+                    except KeyError:
+                        tmpfilename = os.path.join(self.build_temp,sources[i][1:])
+                        if os.path.isfile       (tmpfilename):
+                            filename = tmpfilename
+                        else:
+                            raise DistutilsSetupError(
+                                 'The %s filename declared in the source '
+                                 'files of the program %s have not been '
+                                 'built by the installation process' % (sources[i],
+                                                                         exe_name))
+                    sources[i]=filename
+                    log.info("%s changed to %s",message,filename)
+                else:
+                    log.info("%s ok",message)
     
-        return sources
+            return sources
     
-    def build_extensions(self):
-        # First, sanity-check the 'extensions' list
+        def build_extensions(self):
+            # First, sanity-check the 'extensions' list
             
-        for ext in self.extensions:
-            ext.sources = self.substitute_sources(ext.name,ext.sources)
+            for ext in self.extensions:
+                ext.sources = self.substitute_sources(ext.name,ext.sources)
                 
-        self.check_extensions_list(self.extensions)
+            self.check_extensions_list(self.extensions)
     
-        for ext in self.extensions:
-            log.info("%s :-> %s",ext.name,ext.sources)
-            ext.sources = self.cython_sources(ext.sources, ext)
-            self.build_extension(ext)
+            for ext in self.extensions:
+                log.info("%s :-> %s",ext.name,ext.sources)
+                ext.sources = self.cython_sources(ext.sources, ext)
+                self.build_extension(ext)
     
             
-    def run(self):
-        self.modifyDocScripts()
+        def run(self):
+            self.modifyDocScripts()
     
-        for cmd_name in self.get_sub_commands():
-            self.run_command(cmd_name)
+            for cmd_name in self.get_sub_commands():
+                self.run_command(cmd_name)
     
-        cython_options.annotate = True
-        ori_build_ext.run(self)  # @UndefinedVariable
+            cython_options.annotate = True
+            ori_build_ext.run(self)  # @UndefinedVariable
             
     
-    def has_files(self):
-        return self.distribution.has_files()
+        def has_files(self):
+            return self.distribution.has_files()
     
-    def has_executables(self):
-        return self.distribution.has_executables()
-    
-    sub_commands = [('build_files',has_files),
-                    ('build_cexe', has_executables)
-                    ] + \
-                   ori_build_ext.sub_commands
+        def has_executables(self):
+            return self.distribution.has_executables()
         
+        sub_commands = [('build_files',has_files),
+                        ('build_cexe', has_executables)
+                        ] + \
+                       ori_build_ext.sub_commands
+
+except ImportError:
+    from distutils.command import build_ext  # @UnusedImport
            
 
 

distutils.ext/obidistutils/serenity/bootstrappip.py

@@ -0,0 +1,36 @@
+'''
+Created on 22 janv. 2016
+
+@author: coissac
+'''
+
+import sys
+from urllib import request
+import os.path
+
+from obidistutils.serenity.util import get_serenity_dir
+from obidistutils.serenity.rerun import rerun_with_anothe_python
+from obidistutils.serenity.checkpython import is_a_virtualenv_python
+
+getpipurl="https://bootstrap.pypa.io/get-pip.py"
+
+def bootstrap():
+
+    getpipfile=os.path.join(get_serenity_dir(),"get-pip.py")
+    
+    with request.urlopen(getpipurl) as getpip:
+        with open(getpipfile,"wb") as out:
+            for l in getpip:
+                out.write(l)
+                
+    python = sys.executable
+    
+    if is_a_virtualenv_python():
+        command= "%s %s" % (python,getpipfile)        
+    else:
+        command= "%s %s --user" % (python,getpipfile)
+    
+    os.system(command)
+    
+    rerun_with_anothe_python(python)
+    

distutils.ext/obidistutils/serenity/checkpackage.py

@@ -7,8 +7,13 @@ Created on 2 oct. 2014
 import re
 import os
 
-import pip                                              # @UnresolvedImport
-from pip.utils import get_installed_distributions       # @UnresolvedImport
+try:
+    import pip                                              # @UnresolvedImport
+    from pip.utils import get_installed_distributions       # @UnresolvedImport
+except ImportError:
+    from .bootstrappip import bootstrap
+    bootstrap()
+
 from distutils.version import StrictVersion             # @UnusedImport
 from distutils.errors import DistutilsError
 from distutils import log

doc/.gitignore

@@ -1,3 +1,5 @@
 /build/
 /doxygen/
 /build_dir.txt
+/.DS_Store
+/.gitignore

doc/Makefile

@@ -57,7 +57,7 @@ html:
 	@echo "Generating Doxygen documentation..."
 	doxygen Doxyfile
 	@echo "Doxygen documentation generated. \n"
-	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
+	$(SPHINXBUILD) -b html -c ./ $(ALLSPHINXOPTS) $(BUILDDIR)/html
 	@echo
 	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
 

doc/conf.py

@@ -36,7 +36,7 @@ extensions = [
     'sphinx.ext.pngmath',
     'sphinx.ext.ifconfig',
     'sphinx.ext.viewcode',
-    'breathe',
+#    'breathe',
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -51,7 +51,7 @@ source_suffix = '.rst'
 #source_encoding = 'utf-8-sig'
 
 # The master toctree document.
-master_doc = 'index'
+master_doc = 'source/index'
 
 # General information about the project.
 project = u'OBITools3'
@@ -292,7 +292,7 @@ texinfo_documents = [
 #texinfo_no_detailmenu = False
 
 #Breathe configuration
-sys.path.append( "../breathe/" )
-breathe_projects = { "OBITools3": "../doxygen/xml/" }
+sys.path.append( "breathe/" )
+breathe_projects = { "OBITools3": "doxygen/xml/" }
 breathe_default_project = "OBITools3"
 

doc/doxygen/xml/.gitignore

@@ -1,4 +0,0 @@
-# Ignore everything in this directory
-*
-# Except this file
-!.gitignore

doc/source/DMS.rst

@@ -13,7 +13,7 @@ Up to now, each of these categories of data were stored in separate
 files, and nothing made it mandatory to keep them together.
 
 
-The `Data Management System` (DMS) of OBITools3 can be regarded as a basic 
+The `Data Management System` (DMS) of OBITools3 can be viewed like a basic 
 database system.
 
 
@@ -27,9 +27,7 @@ OBIDMS UML
 
 An OBIDMS directory contains :
   * one `OBIDMS history file <#obidms-history-files>`_
-  * Two different kinds of directories :
-     * OBIDMS column directories
-     * OBIDMS column group directories containing OBIDMS column directories
+  * OBIDMS column directories
 
 
 OBIDMS column directories
@@ -39,16 +37,9 @@ OBIDMS column directories contain :
     * all the different versions of one OBIDMS column, under the form of different files (`OBIDMS column files <#obidms-column-files>`_)
     * one `OBIDMS version file <#obidms-version-files>`_
 
-The directory name is the column attribute, or sub-attribute if the column directory is in a column group directory.
+The directory name is the column attribute with the extension ``.obicol``.
 
-
-OBIDMS column group directories
-===============================
-
-OBIDMS column group directories contain OBIDMS column directories. They are used to store dictionary-like data, where 
-each key corresponds to an OBIDMS column.
-
-The directory name is the dictionary attribute. Each key is considered a sub-attribute and is associated to its column.
+Example: ``count.obicol``
 
 
 OBIDMS column files
@@ -57,7 +48,7 @@ OBIDMS column files
 Each OBIDMS column file contains :
 	* a header of a size equal to a multiple of PAGESIZE (PAGESIZE being equal to 4096 bytes
 	  on most systems) containing metadata
-	* one column of data with the same `OBIType <types.html#obitypes>`_
+	* Lines of data with the same `OBIType <types.html#obitypes>`_
 
 
 Header
@@ -79,7 +70,14 @@ The header of an OBIDMS column contains :
 Data
 ----
 
-A column of data with the same `OBIType <types.html#obitypes>`_.
+A line of data corresponds to a vector of elements. Each element is associated with an element name. 
+Elements names are stored in the header. The correspondance between an element and its name is done 
+using their order in the lists of elements and elements names.  This structure allows the storage of 
+dictionary-like data.
+
+Example: In the header, the attribute ``elements_names`` will be associated with the value ``"sample_1;
+sample_2;sample_3"``, and a line of data with the type ``OBInt_t`` will be stored as an ``OBInt_t`` vector 
+of size three e.g. ``5|8|4``.
 
 
 Mandatory columns
@@ -158,3 +156,5 @@ operations ever done in the OBIDMS directory and the views in between them :
 .. image:: ./images/history.png
     :width: 150 px
     :align: center
+
+

doc/source/data.rst

@@ -2,8 +2,8 @@
 Data in OBITools3
 #################
 
-The OBITools3 inaugure a new way to manage DNA metabarcoding data. 
-They rely on a `Data management System` (DMS) that can be considered as
+The OBITools3 introduce a new way to manage DNA metabarcoding data. 
+They rely on a `Data management System` (DMS) that can be viewed like 
 a simplified database system.
 
 

doc/source/guidelines.rst

@@ -70,7 +70,7 @@ Tickets should always be labeled with the branches for which they are relevant.
 Documentation
 *************
 
-C functions are documented in the header files.
+C functions are documented in the header files for public functions, and in the source file for private functions.
 
 
 **************
@@ -92,7 +92,7 @@ C99 :
 	* Object layer
 	* OBITools3 library
 
-`Python 3 <https://www.python.org/>`_ :
+`Python 3.5 <https://www.python.org/>`_ :
 	* Top layer code (scripts)	
 	
 For the documentation, `Sphinx <http://sphinx-doc.org/>`_ should be used for both the original 
@@ -111,6 +111,8 @@ Enum members, macros, constants:                ``ALL_CAPS``
 
 Functions, local variables:                     ``lower_case``
 
+Public functions:                               ``obi_lower_case``
+
 Functions that shouldn't be called directly:    ``_lower_case`` (``_`` prefix)
 
 Global variables:                                ``g_lower_case`` (``g_`` prefix)
@@ -120,9 +122,6 @@ Pointers:                                        ``pointer_ptr`` (``_ptr`` suffi
 .. note::
     Underscores are used to delimit 'words'.
 
-.. todo::
-    ``obi_function`` for public functions names?
-
 
 *****************
 Programming rules

doc/source/index.rst

@@ -11,7 +11,6 @@ OBITools3 documentation
 
    Programming guidelines <guidelines>
    Data structures <data>
-   Pistes de reflexion <pistes>
 
 
 Indices and tables

doc/source/specialvalues.rst

@@ -7,13 +7,16 @@ NA values
 =========
 
 All OBITypes have an associated NA (Not Available) value. 
-NA values are implemented by specifying an explicit NA value for each type, corresponding to the R standards:
+NA values are implemented by specifying an explicit NA value for each type, 
+corresponding to the R standards as much as possible:
 
-* For the types ``OBIInt_t``, ``OBIBool_t``, ``OBIIdx_t`` and ``OBITaxid_t``, the NA value is ``INT_MIN``.
+* For the type ``OBIInt_t``, the NA value is ``INT_MIN``.
 
-* For the type ``OBIChar_t``: the NA value is ``\0`` (?).
+* For the type ``OBIBool_t``, the NA value is ``2``.
 
-* For the type ``OBIStr_t`` : the NA value is ``\0`` (?).
+* For the type ``OBIIdx_t`` and ``OBITaxid_t``, the NA value is ``SIZE_MAX``.
+
+* For the type ``OBIChar_t``: the NA value is ``\0``.
 
 * For the type ``OBIFloat_t``::
 

doc/source/types.rst

@@ -7,11 +7,14 @@ OBITypes
 :download:`html version of the OBITypes UML file <UML/OBITypes_UML.class.violet.html>`
 
 
+.. image:: ./UML/Obicolumn_classes_UML.png
+
+:download:`html version of the OBIDMS classes UML file <UML/Obicolumn_classes_UML.class.violet.html>`
+
+
 .. toctree::
    :maxdepth: 2
 
    The elementary types <elementary>
    The containers <containers>
    Special values <specialvalues>
-
-

doc/sphinx/build_dir.txt

@@ -0,0 +1 @@
+build/lib.macosx-10.6-intel-3.5

doc/sphinx/source/pistes.rst

@@ -1,23 +0,0 @@
-###################
-Pistes de reflexion
-###################
-
-
-******************************
-Ce que l'on veut pouvoir faire
-******************************
-
-    * Gerer les valeurs manquantes
-    * Modifier une colonne en cours d'ecriture (mmap)
-    * Ajouter des valeurs a la fin du fichier d'une colonne en cours d'ecriture (mmap)
-    
-
-******
-Divers
-******
-
-* Si l'ordre d'une colonne est change, elle est reecrite (pas d'index).
-* Utilisation de semaphores pour la lecture
-* Utilisation de tas pour l'indexation des chaines de caracteres. Chaque colonne dont 
-  le type est OBIStr_t est stockee dans 3 fichiers : un fichier contenant les chaines, un 
-  fichier contenant les index, et un fichier contenant le tas.

python/obi.py

@@ -0,0 +1,50 @@
+#!/usr/local/bin/python3.4
+'''
+obi -- shortdesc
+
+obi is a description
+
+It defines classes_and_methods
+
+@author:     user_name
+
+@copyright:  2014 organization_name. All rights reserved.
+
+@license:    license
+
+@contact:    user_email
+@deffield    updated: Updated
+'''
+
+
+default_config = { 'software'       : "The OBITools",
+                   'log'            : False,
+                   'loglevel'       : 'INFO',
+                   'progress'       : True,
+                   'defaultdms'     : None
+                  }
+
+root_config_name='obi'
+
+from obitools3.apps.config import getConfiguration     # @UnresolvedImport
+from obitools3.version import version
+
+__all__     = []
+__version__ = version
+__date__    = '2014-09-28'
+__updated__ = '2014-09-28'
+
+DEBUG = 1
+TESTRUN = 0
+PROFILE = 0
+
+
+if __name__ =="__main__":
+    
+    config = getConfiguration(root_config_name,
+                              default_config)    
+                
+    config[root_config_name]['module'].run(config)
+    
+    
+    

python/obitools3/apps/arguments.pxd

@@ -0,0 +1,3 @@
+#cython: language_level=3
+
+cpdef buildArgumentParser(str configname, str softname)

python/obitools3/apps/arguments.pyx

@@ -0,0 +1,61 @@
+#cython: language_level=3
+
+'''
+Created on 27 mars 2016
+
+@author: coissac
+'''
+
+import argparse
+import sys
+
+
+from .command import getCommandsList
+
+class ObiParser(argparse.ArgumentParser): 
+    def error(self, message):
+        sys.stderr.write('error: %s\n' % message)
+        self.print_help()
+        sys.exit(2)
+
+cpdef buildArgumentParser(str configname, 
+                          str softname):
+    parser = ObiParser()
+    
+    parser.add_argument('--version',   dest='%s:version' % configname, 
+                                       action='store_true', 
+                                       default=False, 
+                        help='Print the version of %s' % softname)
+
+    parser.add_argument('--log',       dest='%s:log' % configname, 
+                                       action='store',
+                                       type=str,
+                                       default=None, 
+                        help='Create a logfile')
+
+    parser.add_argument('--no-progress', dest='%s:progress' % configname, 
+                                       action='store_false', 
+                                       default=None, 
+                        help='Do not print the progress bar during analyzes')
+
+    subparsers = parser.add_subparsers(title='subcommands',
+                                       description='valid subcommands',
+                                       help='additional help')
+    
+    commands = getCommandsList()
+    
+    for c in commands:
+        module = commands[c]
+        
+        if hasattr(module, "run"):
+            if hasattr(module, "__title__"):
+                sub = subparsers.add_parser(c,help=module.__title__)
+            else:
+                sub = subparsers.add_parser(c)
+    
+            if hasattr(module, "addOptions"):
+                module.addOptions(sub)
+            
+            sub.set_defaults(**{'%s:module'  % configname : module})
+                        
+    return parser

python/obitools3/apps/command.pxd

@@ -0,0 +1,3 @@
+#cython: language_level=3
+
+cdef object loadCommand(str name,loader)

python/obitools3/apps/command.pyx

@@ -0,0 +1,44 @@
+#cython: language_level=3
+
+'''
+Created on 27 mars 2016
+
+@author: coissac
+'''
+
+import pkgutil
+
+from obitools3 import commands
+
+cdef object loadCommand(str name,loader):
+    '''
+    Load a command module from its name and an ImpLoader
+    
+    This function is for internal use
+    
+    @param name:   name of the module
+    @type name: str 
+    @param loader: the module loader
+    @type loader: ImpLoader
+    
+    @return the loaded module
+    @rtype: module 
+    '''
+    
+    module = loader.find_module(name).load_module(name)
+    return module
+
+def getCommandsList():
+    '''
+    Returns the list of sub-commands available to the main `obi` command
+    
+    @return: a dict instance with key corresponding to each command and
+             value corresponding to the module
+             
+    @rtype: dict
+    '''
+    
+    cdef dict cmds = dict((x[1],loadCommand(x[1],x[0])) 
+                           for x in pkgutil.iter_modules(commands.__path__) 
+                           if not x[2])
+    return cmds

python/obitools3/apps/config.pxd

@@ -0,0 +1,10 @@
+#cython: language_level=3
+
+cpdef str setRootConfigName(str rootname)
+cpdef str getRootConfigName()
+
+cdef dict buildDefaultConfiguration(str root_config_name,
+                                    dict  config)
+
+cpdef dict getConfiguration(str root_config_name=?,
+                                    dict  config=?)

python/obitools3/apps/config.pyx

@@ -0,0 +1,103 @@
+#cython: language_level=3
+
+'''
+Created on 27 mars 2016
+
+@author: coissac
+'''
+
+import sys
+
+from .command   import  getCommandsList
+from .logging   cimport getLogger
+from .arguments cimport buildArgumentParser
+
+from ..version import version
+from _curses import version
+
+cdef dict __default_config__ = {}
+
+
+cpdef str setRootConfigName(str rootname):
+    global __default_config__
+    if '__root_config__' in __default_config__:
+        if __default_config__["__root_config__"] in __default_config__:
+            __default_config__[rootname]=__default_config__[__default_config__["__root_config__"]]
+            del __default_config__[__default_config__["__root_config__"]]
+    __default_config__['__root_config__']=rootname
+    return rootname
+
+cpdef str getRootConfigName():
+    global __default_config__
+    return __default_config__.get('__root_config__',None)
+
+cdef dict buildDefaultConfiguration(str root_config_name,
+                                    dict  config):
+    global __default_config__
+    
+    __default_config__.clear()
+    setRootConfigName(root_config_name)    
+
+    __default_config__[root_config_name]=config
+    
+    config['version']=version
+    
+    commands = getCommandsList()
+    
+    for c in commands:
+        module = commands[c]
+    
+        assert hasattr(module, "run")
+        
+        if hasattr(module, 'default_config'):
+            __default_config__[c]=module.default_config
+        else:
+            __default_config__[c]={}
+                        
+    return __default_config__
+        
+
+cpdef dict getConfiguration(str root_config_name="__default__",
+                            dict  config={}):
+    global __default_config__
+
+    if '__done__' in __default_config__:
+        return __default_config__
+    
+    if root_config_name=="__default__":
+        raise RuntimeError("No root_config_name specified")
+    
+    if not config:
+        raise RuntimeError("Base configuration is empty")
+    
+    
+    
+    config =  buildDefaultConfiguration(root_config_name,
+                                        config)
+    
+    parser = buildArgumentParser(root_config_name,
+                                 config[root_config_name]['software'])
+    
+    options = vars(parser.parse_args())
+
+    if options['%s:version' % root_config_name]:
+        print("%s - Version %s" % (config[root_config_name]['software'],
+                                   config[root_config_name]['version']))
+        sys.exit(0)
+    
+    for k in options:
+        section,key = k.split(':')
+        s = config[section]
+        if options[k] is not None:
+            s[key]=options[k]
+            
+    if not 'module' in config[root_config_name]:
+        print('\nError: No command specified',file=sys.stderr)
+        parser.print_help()
+        sys.exit(2)
+                
+    getLogger(config)
+    
+    config['__done__']=True
+            
+    return config

python/obitools3/apps/logging.pxd

@@ -0,0 +1,3 @@
+#cython: language_level=3
+
+cpdef getLogger(dict config)

python/obitools3/apps/logging.pyx

@@ -0,0 +1,46 @@
+#cython: language_level=3
+
+'''
+Created on 27 mars 2016
+
+@author: coissac
+'''
+
+import logging
+import sys
+
+cpdef getLogger(dict config):
+    '''
+    Returns the logger as defined by the command line option
+    or by the config file
+    :param config:
+    '''
+    
+    root = config["__root_config__"]
+ 
+    level  = config[root]['loglevel'] 
+    logfile= config[root]['log'] 
+    
+    rootlogger   = logging.getLogger()
+    logFormatter = logging.Formatter("%(asctime)s [%(levelname)-5.5s]  %(message)s")
+
+    stderrHandler = logging.StreamHandler(sys.stderr)
+    stderrHandler.setFormatter(logFormatter)
+
+    rootlogger.addHandler(stderrHandler)
+    
+    if logfile:
+        fileHandler = logging.FileHandler(logfile)
+        fileHandler.setFormatter(logFormatter)
+        rootlogger.addHandler(fileHandler)
+    
+    try:
+        loglevel = getattr(logging, level) 
+    except:
+        loglevel = logging.INFO
+        
+    rootlogger.setLevel(loglevel)
+    
+    config[root]['logger']=rootlogger
+    
+    return rootlogger

python/obitools3/apps/progress.pxd

@@ -0,0 +1,66 @@
+#cython: language_level=3
+
+from ..utils cimport str2bytes
+
+cdef extern from "stdio.h":
+    struct FILE
+    int fprintf(FILE *stream, char *format, ...)
+    int fputs(char *string, FILE *stream)
+    FILE* stderr
+    ctypedef unsigned int off_t "unsigned long long"
+    
+cdef extern from "unistd.h":
+    int fsync(int fd);
+
+cdef extern from "time.h":
+    struct tm :
+        int tm_yday 
+        int tm_hour
+        int tm_min
+        int tm_sec
+
+    enum: CLOCKS_PER_SEC
+    
+    ctypedef int time_t
+    ctypedef int clock_t
+    ctypedef int suseconds_t
+    
+    struct timeval:
+        time_t      tv_sec     #  seconds */
+        suseconds_t tv_usec    #  microseconds */
+
+
+    struct timezone :
+        int tz_minuteswest;    # minutes west of Greenwich
+        int tz_dsttime;        # type of DST correction 
+
+        
+    int gettimeofday(timeval *tv, timezone *tz)
+
+
+    tm *gmtime_r(time_t *clock, tm *result)
+    time_t time(time_t *tloc)
+    clock_t clock()
+
+cdef class ProgressBar:
+    cdef off_t   maxi
+    cdef clock_t starttime
+    cdef clock_t lasttime
+    cdef clock_t tickcount
+    cdef int freq
+    cdef int cycle
+    cdef int arrow
+    cdef int lastlog
+    cdef bint ontty
+    cdef int fd
+    
+    cdef bytes head
+    cdef char *chead
+    
+    cdef object logger
+
+    cdef char  *wheel
+    cdef char  *spaces
+    cdef char*  diese 
+    
+    cdef clock_t clock(self)

python/obitools3/apps/progress.pyx

@@ -0,0 +1,138 @@
+#cython: language_level=3
+
+'''
+Created on 27 mars 2016
+
+@author: coissac
+'''
+
+import sys
+from ..utils cimport bytes2str
+
+cdef class ProgressBar:
+    cdef clock_t clock(self):
+        cdef clock_t t
+        cdef timeval tp
+        cdef clock_t s
+        
+        <void> gettimeofday(&tp,NULL)
+        s = <clock_t> (<double> tp.tv_usec * 1.e-6 * <double> CLOCKS_PER_SEC)
+        t = tp.tv_sec * CLOCKS_PER_SEC + s 
+        
+        return t
+
+    def __init__(self,
+                 off_t maxi,
+                 dict  config,
+                 str head="",
+                 double seconde=0.1):
+        self.starttime = self.clock()
+        self.lasttime  = self.starttime
+        self.tickcount = <clock_t> (seconde * CLOCKS_PER_SEC)
+        self.freq      = 1
+        self.cycle     = 0
+        self.arrow     = 0
+        self.lastlog   = 0
+        
+        self.ontty = sys.stderr.isatty()
+        
+        if (maxi<=0):
+            maxi=1
+            
+        self.maxi = maxi
+        self.head = str2bytes(head)
+        self.chead= self.head 
+        
+        
+        self.logger=config[config["__root_config__"]]["logger"]
+        self.wheel =  '|/-\\'
+        self.spaces='          ' \
+                    '          ' \
+                    '          ' \
+                    '          ' \
+                    '          '
+        self.diese ='##########' \
+                    '##########' \
+                    '##########' \
+                    '##########' \
+                    '##########'  
+                           
+    def __call__(self,object pos):
+        cdef off_t    ipos
+        cdef clock_t  elapsed
+        cdef clock_t  newtime
+        cdef clock_t  delta
+        cdef clock_t  more 
+        cdef double   percent 
+        cdef tm remain
+        cdef int days,hour,minu,sec
+        cdef off_t fraction
+        cdef int twentyth
+        
+        self.cycle+=1
+    
+        if self.cycle % self.freq == 0:
+            self.cycle=1
+            newtime  = self.clock()
+            delta         = newtime - self.lasttime
+            self.lasttime = newtime
+            elapsed       = newtime - self.starttime
+#            print(" ",delta,elapsed,elapsed/CLOCKS_PER_SEC,self.tickcount)
+            
+            if   delta < self.tickcount / 5 :
+                self.freq*=2
+            elif delta > self.tickcount * 5 and self.freq>1:
+                self.freq/=2
+                
+            
+            if callable(pos):
+                ipos=pos()
+            else:
+                ipos=pos
+                
+            if ipos==0:
+                ipos=1                
+
+            percent = <double>ipos/<double>self.maxi
+            more = <time_t>((<double>elapsed / percent * (1. - percent))/CLOCKS_PER_SEC)
+            <void>gmtime_r(&more, &remain)
+            days  = remain.tm_yday 
+            hour  = remain.tm_hour
+            minu  = remain.tm_min
+            sec   = remain.tm_sec
+                
+            if self.ontty:
+                fraction=<int>(percent * 50.)
+                self.arrow=(self.arrow+1) % 4
+                self.diese[fraction]=0
+                self.spaces[50 - fraction]=0
+            
+                if days:
+                    <void>fprintf(stderr,b'\r%s %5.1f %% |%s%c%s] remain : %d days %02d:%02d:%02d',
+                                    self.chead,
+                                    percent*100,
+                                    self.diese,self.wheel[self.arrow],self.spaces,
+                                    days,hour,minu,sec)
+                else:
+                    <void>fprintf(stderr,b'\r%s %5.1f %% |%s%c%s] remain : %02d:%02d:%02d',
+                                    self.chead,
+                                    percent*100.,
+                                    self.diese,self.wheel[self.arrow],self.spaces,
+                                    hour,minu,sec)
+                self.diese[fraction]=b'#'
+                self.spaces[50 - fraction]=b' '
+
+            twentyth = int(percent * 20)
+            if twentyth != self.lastlog:
+                
+                if self.ontty:
+                    <void>fputs(b'\n',stderr)
+                    
+                self.logger.info('%s %5.1f %% remain : %02d:%02d:%02d' % (
+                                        bytes2str(self.head),
+                                        percent*100.,
+                                        hour,minu,sec))
+                self.lastlog=twentyth
+        else:
+            self.cycle+=1
+

python/obitools3/commands/count.py

@@ -0,0 +1,44 @@
+'''
+Created on 8 mars 2016
+
+@author: coissac
+'''
+
+from obitools3.apps.progress import ProgressBar  # @UnresolvedImport
+import time
+
+__title__="Counts sequences in a sequence set"
+
+
+default_config = {   'countmode'       : None
+                 }
+
+def addOptions(parser):
+    parser.add_argument(dest='obi:input',     metavar='obi:input', 
+                                              nargs='?', 
+                                              default=None,
+                        help='input data set' )
+
+    group=parser.add_argument_group('Obicount specific options')
+    group.add_argument('-s','--sequence',
+                             action="store_true", dest="count:sequence",
+                             default=False,
+                             help="Prints only the number of sequence records."
+                             )
+ 
+    group.add_argument('-a','--all',
+                             action="store_true", dest="count:all",
+                             default=False,
+                             help="Prints only the total count of sequence records (if a sequence has no `count` attribute, its default count is 1) (default: False)."
+                             )
+
+    
+    
+def run(config):
+    # The code of my command
+    pb = ProgressBar(1000,config,seconde=1)
+    
+    for i in range(1,1001):
+        pb(i)
+        time.sleep(0.01)
+        

python/obitools3/commands/import.pyx

@@ -0,0 +1,139 @@
+from obitools3.apps.progress cimport ProgressBar  # @UnresolvedImport
+from obitools3.files.universalopener cimport uopen
+from obitools3.parsers.fasta import fastaIterator
+from obitools3.parsers.fastq import fastqIterator
+from obitools3.obidms._obidms import OBIDMS
+
+
+import time
+
+__title__="Counts sequences in a sequence set"
+
+
+default_config = {   'destview'     : None,
+                     'skip'         : 0,
+                     'only'         : None,
+                     'skiperror'    : False,
+                     'seqinformat'  : None,
+                     'moltype'      : 'nuc',
+                     'filename'     : None
+                 }
+
+def addOptions(parser):
+    parser.add_argument(dest='import:filename',     
+                        metavar='<FILENAME>', 
+                        nargs='?', 
+                        default=None,
+                        help='sequence file name to be imported' )
+
+    group=parser.add_argument_group('obi import specific options')
+
+    group.add_argument('--default-dms','-d',
+                     action="store", dest="obi:defaultdms",
+                     metavar='<DMS NAME>',
+                     default=None,
+                     type=str,
+                     help="Name of the default DMS for reading and writing data")
+    
+    group.add_argument('--destination-view','-v',
+                     action="store", dest="import:destview",
+                     metavar='<VIEW NAME>',
+                     default=None,
+                     type=str,
+                     required=True,
+                     help="Name of the default DMS for reading and writing data")
+
+    group=parser.add_argument_group('obi import specific options')
+    
+    group.add_argument('--skip',
+                     action="store", dest="import:skip",
+                     metavar='<N>',
+                     default=None,
+                     type=int,
+                     help="skip the N first sequences")
+
+    group.add_argument('--only',
+                     action="store", dest="import:only",
+                     metavar='<N>',
+                     default=None,
+                     type=int,
+                     help="treat only N sequences")
+
+    group.add_argument('--skip-on-error',
+                     action="store_true", dest="import:skiperror",
+                     default=None,
+                     help="Skip sequence entries with parse error")
+    
+    group.add_argument('--fasta',
+                     action="store_const", dest="import:seqinformat",
+                     default=None,
+                     const='fasta',
+                     help="Input file is in fasta nucleic format (including obitools fasta extentions)")
+
+    group.add_argument('--fastq',
+                     action="store_const", dest="import:seqinformat",
+                     default=None,
+                     const='fastq',
+                     help="Input file is in sanger fastq nucleic format (standard fastq)")
+
+    group.add_argument('--nuc',
+                     action="store_const", dest="import:moltype",
+                     default=None,
+                     const='nuc',
+                     help="Input file contains nucleic sequences")
+    
+    group.add_argument('--prot',
+                     action="store_const", dest="import:moltype",
+                     default=None,
+                     const='pep',
+                     help="Input file contains protein sequences")
+
+    
+    
+def run(config):
+    pb = ProgressBar(35000000,config,seconde=5)
+      
+    inputs = uopen(config['import']['filename'])
+    
+    get_quality = False
+    if config['import']['seqinformat']=='fasta':
+        iseq = fastaIterator(inputs)
+        view_type="NUC_SEQS_VIEW"
+    elif config['import']['seqinformat']=='fastq':
+        iseq = fastqIterator(inputs)
+        view_type="NUC_SEQS_VIEW"
+        get_quality = True
+    else:
+        raise RuntimeError('No file format specified')
+    
+    # Temporary way to handle NA values
+    #NA_list = ["nan"]
+    
+    # Create DMS
+    d = OBIDMS(config['obi']['defaultdms'])
+    
+    # Create view
+    view = d.new_view(config['import']['destview'], view_type=view_type)
+    
+    i = 0
+    for seq in iseq:
+        pb(i)
+        view[i].set_id(seq['id'])
+        view[i].set_definition(seq['definition'])
+        view[i].set_sequence(seq['sequence'])
+        if get_quality :
+            view[i].set_quality(seq['quality'])
+        for tag in seq['tags'] :
+            #print(tag, seq['tags'][tag])
+            #if seq['tags'][tag] not in NA_list :
+            view[i][tag] = seq['tags'][tag]
+        i+=1
+
+    #print(i)
+    print(view.__repr__())
+
+    view.save_and_close()
+    d.close()
+    
+    print("Done.")
+        

python/obitools3/files/linebuffer.pxd

@@ -0,0 +1,6 @@
+#cython: language_level=3
+
+
+cdef class LineBuffer:
+    cdef object fileobj
+    cdef int    size

python/obitools3/files/linebuffer.pyx

@@ -0,0 +1,23 @@
+#cython: language_level=3
+
+'''
+Created on 30 mars 2016
+
+@author: coissac
+'''
+
+cdef class LineBuffer:
+
+    def __init__(self,object fileobj,int size=100000000):
+        self.fileobj=fileobj 
+        self.size=size
+    
+    def __iter__(self):
+        cdef list buff = self.fileobj.readlines(self.size)
+        cdef str  l
+        
+        while buff:
+            for l in buff:
+                yield l
+            buff = self.fileobj.readlines(self.size)
+    

python/obitools3/files/uncompress.pxd

@@ -0,0 +1,18 @@
+#cython: language_level=3
+
+cdef class MagicKeyFile:
+    cdef object stream
+    cdef str    stream_mode
+    cdef object binary
+    cdef bytes  key 
+    cdef int    keylength 
+    cdef int    pos
+    
+    cpdef bytes read(self,int size=?)
+    cpdef bytes read1(self,int size=?)
+    cpdef int tell(self)
+
+    
+cdef class CompressedFile:
+    cdef object accessor
+    

python/obitools3/files/uncompress.pyx

@@ -0,0 +1,117 @@
+#cython: language_level=3
+
+'''
+Created on 28 mars 2016
+
+@author: coissac
+'''
+
+import zipfile
+import bz2
+import gzip
+
+import io
+
+cdef class MagicKeyFile:
+    def __init__(self,stream,length=2):
+        
+        binary=stream
+        self.stream = stream
+        self.stream_mode = None 
+        if hasattr(stream, "mode"):
+            self.stream_mode = stream.mode
+            if (not 'b' in stream.mode and 
+                hasattr(stream, "buffer") and 
+                'b' in stream.buffer.mode):
+                binary=stream.buffer
+
+        if (self.stream_mode is None and 
+            not (hasattr(stream, 'headers') and 
+                 hasattr(stream.headers, "keys") and
+                 'Content-type' in stream.headers)):
+            raise TypeError("stream does not present the good interface")
+        
+        self.binary=binary
+        self.key=binary.read(length)
+        self.keylength=length
+        self.pos=0
+        
+    cpdef bytes read(self,int size=-1):
+        cdef bytes r
+        
+        if self.pos < self.keylength:
+            if size > (self.keylength - self.pos):
+                size = size - self.keylength + self.pos
+                r = self.key[self.pos:] + self.binary.read(size)
+                self.pos=self.keylength + 1
+            elif size >=0 :
+                r = self.key[self.pos:(self.pos+size)]
+                self.pos+=size
+            else:
+                r = self.key[self.pos:] + self.binary.read(size)
+                self.pos=self.keylength + 1
+        else:
+            r = self.binary.read(size)
+            
+        return r
+
+    cpdef bytes read1(self,int size=-1):
+        return self.read(size)
+
+    cpdef int tell(self):
+        cdef int p
+        
+        if self.pos < self.keylength:
+            p = self.pos 
+        else:
+            p = self.binary.tell()
+        
+        return p
+    
+    def __getattr__(self,name):
+        return getattr(self.binary, name)
+        
+
+    
+cdef class CompressedFile:
+    
+    
+    def __init__(self,stream):
+        cdef int            keylength
+        cdef MagicKeyFile   magic
+        cdef str            compressor
+        cdef bytes          k 
+        cdef object         c
+        
+        cdef dict compress = { 'zip' : (b'\x50\x4b\x03\x04',zipfile.ZipFile),
+                               'bz2' : (b'\x42\x5a\x68',bz2.BZ2File),
+                               'gz'  : (b'\x1f\x8b\x08',gzip.open)
+                   }
+        
+        keylength = max([len(x[0]) for x in compress.values()])
+        magic=MagicKeyFile(stream,keylength)
+        
+        self.accessor = None
+        
+        for compressor in compress:
+            k,c = compress[compressor]
+            if magic.key.startswith(k):
+                self.accessor = c(magic)
+        
+        if self.accessor is None:
+            self.accessor = magic
+        
+        if ((hasattr(stream, 'headers') and 
+             hasattr(stream.headers, "keys") and
+             'Content-type' in stream.headers and
+             stream.headers['Content-type'].startswith('text/')) or
+            'b' not in magic.stream_mode):
+            self.accessor = io.TextIOWrapper(self.accessor)
+        
+    
+    def __getattr__(self,name):
+        return getattr(self.accessor, name)
+    
+    def __iter__(self):
+        for x in self.accessor:
+            yield x 

python/obitools3/files/universalopener.pxd

@@ -0,0 +1,5 @@
+#cython: language_level=3
+
+from .uncompress cimport CompressedFile
+
+cpdef CompressedFile uopen(str name, mode=?)

python/obitools3/files/universalopener.pyx

@@ -0,0 +1,23 @@
+#cython: language_level=3
+
+'''
+Created on 25 mars 2016
+
+@author: coissac
+'''
+
+from urllib.request import urlopen
+
+
+cpdef CompressedFile uopen(str name, mode='r'):
+    cdef CompressedFile c
+    
+    try:
+        f = urlopen(name)
+    except:
+        f = open(name,mode) 
+        
+    c = CompressedFile(f)
+    
+    return c
+    

python/obitools3/format/fasta.pxd

@@ -0,0 +1,10 @@
+from ..utils cimport bytes2str
+from .header cimport HeaderFormat
+from cython.view cimport array as cvarray
+
+cdef class FastaFormat:
+
+    cdef HeaderFormat headerFormater
+    
+    cdef size_t  sequenceBufferLength
+    cdef char*   sequenceBuffer

python/obitools3/format/fasta.pyx

@@ -0,0 +1,32 @@
+cimport cython
+from libc.stdlib cimport malloc, free, realloc
+from libc.string cimport strncpy
+
+cdef class FastaFormat:
+	
+	def __init__(self, list tags=[], bint printNAKeys=False):
+		self.headerFormater = HeaderFormat(True,
+										   tags,
+										   printNAKeys)
+		
+	@cython.boundscheck(False)	
+	def __call__(self, dict data):
+		cdef bytes brawseq      = data['sequence']
+		cdef size_t lseq        = len(brawseq)
+		cdef size_t k=0
+		cdef list   lines       = []
+		
+		for k in range(0,lseq,60):
+			lines.append(brawseq[k:(k+60)])
+			
+		brawseq = b'\n'.join(lines)
+		
+		return "%s\n%s" % (self.headerFormater(data),bytes2str(brawseq))
+					
+		
+				
+			
+		
+		
+		
+		

python/obitools3/format/header.pxd

@@ -0,0 +1,7 @@
+cdef class HeaderFormat:
+
+    cdef str    start
+    cdef set    tags
+    cdef bint   printNaKeys
+    cdef size_t headerBufferLength
+    

python/obitools3/format/header.pyx

@@ -0,0 +1,60 @@
+
+cdef class HeaderFormat:
+    
+    def __init__(self, bint fastaHeader=True, list tags=[], bint printNAKeys=False):
+        '''
+        
+            @param fastaHeader: 
+            @type  fastaHeader: `bool`
+            
+            @param tags:
+            @type  tags: `list` of `bytes`
+            
+            @param printNAKeys: 
+            @type  printNAKeys: `bool` 
+        '''
+        
+        self.tags           = set(tags)
+        self.printNaKeys    = printNAKeys
+        
+        if fastaHeader:
+            self.start=">"
+        else:
+            self.start="@"
+        
+        self.headerBufferLength = 1000
+        #self.headerBuffer = []
+        
+    def __call__(self, dict data):
+        cdef str header
+        cdef dict tags = data['tags']
+        cdef set  ktags
+        cdef list lines = [""]
+        cdef str tagline
+        
+        if self.tags is not None and self.tags:
+            ktags = self.tags
+        else:
+            ktags = set(tags.keys())
+            
+        for k in ktags:
+            if k in tags:
+                value = tags[k]
+                if value is not None or self.printNaKeys:
+                    lines.append("%s=%s;" % (k,tags[k]))
+                    
+        if len(lines) > 1:
+            tagline=" ".join(lines)
+        else:
+            tagline=""
+            
+        if data['definition'] is not None:
+            header = "%s%s%s %s" % (self.start,data['id'],
+                                    tagline,
+                                    data['definition'])
+        else:
+            header = "%s%s%s" % (self.start,data['id'],
+                                 tagline)
+            
+        return header
+        

python/obitools3/obicount.py

@@ -1,31 +0,0 @@
-import sys
-import argparse
-
-
-from obitools3.obidms.obidmscolumn.capidmscolumn import OBIDMS_column
-
-
-if __name__ == '__main__': 
-    
-    parser = argparse.ArgumentParser(description='Computes the sum of a column.')
-    
-    parser.add_argument('-d', '--dms', dest='dms_name', type=str,
-                       help='Name of the OBIDMS containing the column')
-    parser.add_argument('-c', '--column', dest='column_name', type=str, default='count',
-                       help="Name of the OBIDMS column (default: 'count')")
-    parser.add_argument('-v', '--version', dest='version_number', type=int, default=-1,
-                       help='Version number of the column (default: latest version)')
-    
-    args = parser.parse_args()
-   
-    c = OBIDMS_column.open(args.dms_name, args.column_name, version_number=args.version_number)
-    
-    # check that 1 element / line and summable type?
-
-    total = 0    
-    
-    for count in c :
-        total+=count
-    
-    print("Total count = ", total)    
-

python/obitools3/obidms/_obidms.cfiles

@@ -1,5 +1,44 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
 ../../../src/obidms.h
 ../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
@@ -10,5 +49,11 @@
 ../../../src/obilittlebigman.c
 ../../../src/obitypes.h
 ../../../src/obitypes.c
-../../../src/private_at_functions.h
-../../../src/private_at_functions.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
+../../../src/utils.h
+../../../src/utils.c

python/obitools3/obidms/_obidms.pxd

@@ -2,37 +2,94 @@
 
 from .capi.obidms       cimport OBIDMS_p
 from .capi.obidmscolumn cimport OBIDMS_column_p
-from .capi.obitypes     cimport obiversion_t, OBIType_t
+from .capi.obiview      cimport Obiview_p
+from .capi.obitypes     cimport obiversion_t, OBIType_t, index_t
+from ._obitaxo          cimport OBI_Taxonomy
+
+
+cdef class OBIDMS_column:
+
+    cdef  OBIDMS_column_p*   pointer
+    cdef  OBIDMS             dms
+    cdef  OBIView            view
+    cdef  str                data_type
+    cdef  str                dms_name
+    cdef  str                column_name
+    cdef  index_t            nb_elements_per_line
+    cdef  list               elements_names
+        
+    cpdef         update_pointer(self)
+    cpdef list    get_elements_names(self)
+    cpdef str     get_data_type(self)
+    cpdef index_t get_nb_lines_used(self)
+    cpdef str     get_creation_date(self)
+    cpdef str     get_comments(self)
+    cpdef         close(self)
+    @staticmethod
+    cdef  object  get_subclass_type(OBIDMS_column_p column_p)
+
+
+cdef class OBIDMS_column_multi_elts(OBIDMS_column):
+    
+    cpdef set_line(self, index_t line_nb, dict values)
+
+
+cdef class OBIDMS_column_line:
+
+    cdef OBIDMS_column column
+    cdef index_t index
+
+
+cdef class OBIView:
+    
+    cdef Obiview_p pointer
+    cdef str       name
+    cdef str       comments
+    cdef dict      columns
+    cdef OBIDMS    dms
+    
+    cpdef delete_column(self, str column_name)
+    cpdef add_column(self, 
+                     str column_name, 
+                     obiversion_t version_number=*, 
+                     str type=*,
+                     index_t nb_lines=*, 
+                     index_t nb_elements_per_line=*,
+                     list elements_names=*,
+                     str indexer_name=*,
+                     str comments=*,
+                     bint create=*
+                    )
+    cpdef select_line(self, index_t line_nb)
+    cpdef select_lines(self, list line_selection)
+    cpdef save_and_close(self)
+
+
+cdef class OBIView_NUC_SEQS(OBIView):
+
+    cdef OBIDMS_column ids
+    cdef OBIDMS_column sequences
+    cdef OBIDMS_column definitions
+    cdef OBIDMS_column qualities
+    
+    cpdef delete_column(self, str column_name)
+    
+
+cdef class OBIView_line :
+
+    cdef index_t index
+    cdef OBIView view
 
-cdef class OBIDMS_column
 
 cdef class OBIDMS:
 
     cdef OBIDMS_p pointer
     cdef str dms_name
     
-    cpdef dict list(self)
+    cpdef close(self)
+    cpdef OBI_Taxonomy open_taxonomy(self, str taxo_name)
+    cpdef OBIView open_view(self, str view_name)
+    cpdef OBIView new_view(self, str view_name, object view_to_clone=*, list line_selection=*, str view_type=*, str comments=*)
+    cpdef dict read_view_infos(self, str view_name)
+#    cpdef dict read_views(self)    TODO
 
-    cpdef OBIDMS_column open_column(self,
-                                    str column_name,
-                                    bint create=*, 
-                                    bint clone=*, bint clone_data=*,
-                                    obiversion_t version_number=*, 
-                                    OBIType_t data_type=*,
-                                    size_t nb_lines=*, 
-                                    size_t nb_elements_per_line=*, 
-                                    str elements_names=*)
-
-
-cdef class OBIDMS_column:
-
-    cdef OBIDMS_column_p pointer
-    cdef OBIDMS dms
-    cdef str data_type   # TODO keep as OBIType_t? both?
-    cdef str dms_name
-    cdef str column_name
-
-    cpdef object get_item(self, size_t line_nb, str element_name)
-    cpdef list   get_elements_names(self)
-    cpdef str    get_data_type(self)
-    cpdef size_t get_nb_lines_used(self)

python/obitools3/obidms/_obidms.pyx

@@ -1,40 +1,569 @@
 #cython: language_level=3
 
-from pathlib import Path
-
 from obitools3.utils cimport bytes2str, str2bytes
 
-from .capi.obidms cimport obi_dms
-from .capi.obidmscolumn cimport obi_column_get_data_type_from_name, \
-                                obi_column_get_latest_version_from_name, \
-                                obi_column_get_line_count_from_name, \
-                                obi_column_get_nb_lines_used, \
-                                obi_column_get_elements_names, \
-                                obi_create_column, \
-                                obi_clone_column, \
-                                obi_open_column, \
-                                obi_close_column
-from .capi.obitypes cimport const_char_p, name_data_type
+from .capi.obidms cimport obi_dms, \
+                          obi_close_dms
                           
+from .capi.obidmscolumn cimport obi_close_column, \
+                                OBIDMS_column_p, \
+                                OBIDMS_column_header_p
 
-from ._obidms cimport OBIDMS
-from ._obidms cimport OBIDMS_column
+from .capi.obiutils cimport obi_format_date
+                                
+from .capi.obitypes cimport const_char_p, \
+                            OBIType_t, \
+                            OBI_INT, \
+                            OBI_FLOAT, \
+                            OBI_BOOL, \
+                            OBI_CHAR, \
+                            OBI_QUAL, \
+                            OBI_STR, \
+                            OBI_SEQ, \
+                            name_data_type, \
+                            only_ATGC    # discuss
+                            
+from ._obidms cimport OBIDMS, \
+                      OBIDMS_column, \
+                      OBIView, \
+                      OBIView_line
+
+from ._obitaxo cimport OBI_Taxonomy
+
+from ._obiseq cimport OBI_Nuc_Seq, OBI_Nuc_Seq_Stored
 
 from ._obidmscolumn_int cimport OBIDMS_column_int, \
-                                OBIDMS_column_int_writable
+                                OBIDMS_column_multi_elts_int
                                 
 from ._obidmscolumn_float cimport OBIDMS_column_float, \
-                                  OBIDMS_column_float_writable
+                                  OBIDMS_column_multi_elts_float
 
 from ._obidmscolumn_bool cimport OBIDMS_column_bool, \
-                                 OBIDMS_column_bool_writable
+                                 OBIDMS_column_multi_elts_bool
 
 from ._obidmscolumn_char cimport OBIDMS_column_char, \
-                                 OBIDMS_column_char_writable
+                                 OBIDMS_column_multi_elts_char
 
-from ._obidmscolumn_idx cimport OBIDMS_column_idx, \
-                                OBIDMS_column_idx_writable
+from ._obidmscolumn_qual cimport OBIDMS_column_qual, \
+                                 OBIDMS_column_multi_elts_qual
 
+from ._obidmscolumn_str cimport OBIDMS_column_str, \
+                                OBIDMS_column_multi_elts_str
+
+from ._obidmscolumn_seq cimport OBIDMS_column_seq, \
+                                OBIDMS_column_multi_elts_seq
+
+from .capi.obiview cimport Obiview_p, \
+                           Obiview_infos_p, \
+                           Column_reference_p, \
+                           obi_new_view_nuc_seqs, \
+                           obi_new_view, \
+                           obi_new_view_cloned_from_name, \
+                           obi_new_view_nuc_seqs_cloned_from_name, \
+                           obi_view_map_file, \
+                           obi_view_unmap_file, \
+                           obi_open_view,  \
+                           obi_view_delete_column, \
+                           obi_view_add_column, \
+                           obi_view_get_column, \
+                           obi_view_get_column, \
+                           obi_view_get_pointer_on_column_in_view, \
+                           obi_select_line, \
+                           obi_select_lines, \
+                           obi_save_and_close_view, \
+                           VIEW_TYPE_NUC_SEQS, \
+                           NUC_SEQUENCE_COLUMN, \
+                           ID_COLUMN, \
+                           DEFINITION_COLUMN, \
+                           QUALITY_COLUMN
+                           
+from libc.stdlib cimport malloc
+from cpython.pycapsule cimport PyCapsule_New, PyCapsule_GetPointer
+
+
+cdef class OBIDMS_column :
+
+    # Should only be initialized through a subclass
+    def __init__(self, OBIView view, str column_name):
+        
+        cdef OBIDMS_column_p  column_p
+        cdef OBIDMS_column_p* column_pp
+        
+        column_pp = obi_view_get_pointer_on_column_in_view(view.pointer, str2bytes(column_name))
+        column_p = column_pp[0]     # TODO ugly cython dereferencing but can't find better
+        
+        # Fill structure
+        self.pointer = column_pp
+        self.dms = view.dms
+        self.view = view
+        self.data_type = bytes2str(name_data_type((column_p.header).returned_data_type))
+        self.column_name = bytes2str((column_p.header).name)
+        self.nb_elements_per_line = (column_p.header).nb_elements_per_line
+        self.elements_names = (bytes2str((column_p.header).elements_names)).split(';')
+
+    def __setitem__(self, index_t line_nb, object value):
+        self.set_line(line_nb, value)
+
+    def __getitem__(self, index_t line_nb):
+        return self.get_line(line_nb)
+
+    def __len__(self):
+        return (self.pointer)[0].header.lines_used
+    
+    def __sizeof__(self):
+        return ((self.pointer)[0].header.header_size + (self.pointer)[0].header.data_size)
+    
+    def __iter__(self):
+    
+        # Declarations
+        cdef index_t lines_used
+        cdef index_t line_nb
+        
+        # Yield each line
+        lines_used = (self.pointer)[0].header.lines_used
+        for line_nb in range(lines_used):
+            yield self.get_line(line_nb)
+    
+    cpdef update_pointer(self):
+        self.pointer = <OBIDMS_column_p*> obi_view_get_pointer_on_column_in_view(self.view.pointer, str2bytes(self.column_name))
+
+    cpdef list get_elements_names(self):
+        return self.elements_names
+         
+    cpdef str get_data_type(self):
+        return self.data_type
+    
+    cpdef index_t get_nb_lines_used(self):
+        return (self.pointer)[0].header.lines_used
+        
+    cpdef str get_creation_date(self):
+        return bytes2str(obi_format_date((self.pointer)[0].header.creation_date))
+    
+    cpdef str get_comments(self):
+        return bytes2str((self.pointer)[0].header.comments)
+    
+    def __str__(self) :
+        cdef str to_print
+        to_print = ''
+        for line in self :
+            to_print = to_print + str(line) + "\n"
+        return to_print
+
+    def __repr__(self) :
+        return (self.column_name + ", version " + str((self.pointer)[0].header.version) + ", data type: " + self.data_type)
+        
+    cpdef close(self):
+        if obi_close_column((self.pointer)[0]) < 0 :
+            raise Exception("Problem closing a column")
+
+
+    @staticmethod
+    cdef object get_subclass_type(OBIDMS_column_p column_p) :
+        
+        cdef object subclass
+        cdef OBIDMS_column_header_p header
+        cdef OBIType_t col_type
+        cdef bint col_writable
+        cdef bint col_one_element_per_line
+
+        header = column_p.header        
+        col_type = header.returned_data_type
+        col_writable = column_p.writable
+        col_one_element_per_line = ((header.nb_elements_per_line) == 1)
+        
+        if col_type == OBI_INT :
+            if col_one_element_per_line :
+                subclass = OBIDMS_column_int
+            else :
+                subclass = OBIDMS_column_multi_elts_int
+        elif col_type == OBI_FLOAT :
+            if col_one_element_per_line :
+                subclass = OBIDMS_column_float
+            else :
+                subclass = OBIDMS_column_multi_elts_float
+        elif col_type == OBI_BOOL :
+            if col_one_element_per_line :
+                subclass = OBIDMS_column_bool
+            else :
+                subclass = OBIDMS_column_multi_elts_bool
+        elif col_type == OBI_CHAR :
+            if col_one_element_per_line :
+                subclass = OBIDMS_column_char
+            else :
+                subclass = OBIDMS_column_multi_elts_char
+        elif col_type == OBI_QUAL :
+            if col_one_element_per_line :
+                subclass = OBIDMS_column_qual
+            else :
+                subclass = OBIDMS_column_multi_elts_qual
+        elif col_type == OBI_STR :
+            if col_one_element_per_line :
+                subclass = OBIDMS_column_str
+            else :
+                subclass = OBIDMS_column_multi_elts_str
+        elif col_type == OBI_SEQ :
+            if col_one_element_per_line :
+                subclass = OBIDMS_column_seq
+            else :
+                subclass = OBIDMS_column_multi_elts_seq
+        else :
+            raise Exception("Problem with the data type")
+
+        return subclass
+
+    
+######################################################################################################
+
+cdef class OBIDMS_column_multi_elts(OBIDMS_column) :
+    
+    def __getitem__(self, index_t line_nb):
+        return OBIDMS_column_line(self, line_nb)
+
+    cpdef set_line(self, index_t line_nb, dict values):
+        for element_name in values :
+            self.set_item(line_nb, element_name, values[element_name])
+
+######################################################################################################
+
+cdef class OBIDMS_column_line :
+
+    def __init__(self, OBIDMS_column column, index_t line_nb) :
+        self.index = line_nb
+        self.column = column
+        
+    def __getitem__(self, str element_name) :
+        return self.column.get_item(self.index, element_name)
+     
+    def __setitem__(self, str element_name, object value):
+        self.column.set_item(self.index, element_name, value)
+
+    def __contains__(self, str element_name):
+        return (element_name in self.column.elements_names)
+
+    def __repr__(self) :
+        return str(self.column.get_line(self.index))
+
+##########################################
+
+
+cdef class OBIView :
+    
+    def __init__(self, OBIDMS dms, str view_name, bint new=False, object view_to_clone=None, list line_selection=None, str comments=""): 
+                
+        cdef Obiview_p              view = NULL
+        cdef int                    i
+        cdef list                   col_list
+        cdef str                    col_name
+        cdef OBIDMS_column          column
+        cdef OBIDMS_column_p        column_p
+        cdef OBIDMS_column_header_p header
+        cdef index_t*               line_selection_p
+        
+        self.dms = dms
+        
+        # Create the C array for the line selection if needed
+        if line_selection is not None :
+            line_selection_p = <index_t*> malloc((len(line_selection) + 1) * sizeof(index_t))
+            for i in range(len(line_selection)) :
+                line_selection_p[i] = line_selection[i]
+            line_selection_p[len(line_selection)] = -1
+        else :
+            line_selection_p = NULL
+       
+        # Create the view if needed
+        if new :
+            if view_to_clone is not None :
+                if type(view_to_clone) == str :
+                    view = obi_new_view_cloned_from_name(dms.pointer, str2bytes(view_name), str2bytes(view_to_clone), line_selection_p, str2bytes(comments))
+                else :
+                    view = obi_new_view(dms.pointer, str2bytes(view_name), (<OBIView> view_to_clone).pointer, line_selection_p, str2bytes(comments))
+            elif view_to_clone is None :
+                view = obi_new_view(dms.pointer, str2bytes(view_name), NULL, line_selection_p, str2bytes(comments))
+        # Else, open the existing view
+        elif not new :
+            if view_name is not None :
+                view = obi_open_view(dms.pointer, str2bytes(view_name))
+            elif view_name is None :
+                view = obi_open_view(dms.pointer, NULL) # TODO discuss
+        
+        if view == NULL :
+            raise Exception("Error creating/opening a view")
+
+        self.pointer = view
+        self.name = bytes2str(view.infos.name)
+        
+        # Go through columns to build list of corresponding python instances
+        self.columns = {}
+        for i in range(view.infos.column_count) :
+            column_p               = <OBIDMS_column_p> (view.columns)[i]
+            header                 = (column_p).header
+            col_name               = bytes2str(header.name)         
+            subclass               = OBIDMS_column.get_subclass_type(column_p)
+            self.columns[col_name] = subclass(self, col_name)
+
+
+    def __repr__(self) :
+        cdef str s
+        s = str(self.name) + ", " + str(self.comments) + ", " + str(self.pointer.infos.line_count) + " lines\n" 
+        for column_name in self.columns :
+            s = s + self.columns[column_name].__repr__() + '\n'
+        return s
+        
+    
+    cpdef delete_column(self, str column_name) :
+        
+        cdef int i
+        cdef Obiview_p view_p
+        cdef OBIDMS_column column
+        cdef OBIDMS_column_p column_p
+        cdef OBIDMS_column_header_p header
+        cdef str column_n
+ 
+        view = self.pointer
+ 
+        if obi_view_delete_column(view_p, str2bytes(column_name)) < 0 :
+            raise Exception("Problem deleting a column from a view")
+        
+        # Update the dictionaries of column pointers and column objects, and update pointers in column objects (make function?):
+        (self.columns).pop(column_name)
+        
+        for column_n in self.columns :
+            (self.columns[column_n]).update_pointer()
+    
+    
+    cpdef add_column(self, 
+                     str column_name, 
+                     obiversion_t version_number=-1, 
+                     str type='',
+                     index_t nb_lines=0, 
+                     index_t nb_elements_per_line=1,
+                     list elements_names=None,
+                     str indexer_name="",
+                     str comments="",
+                     bint create=True
+                     ) :
+        
+        cdef bytes column_name_b
+        cdef bytes elements_names_b
+        cdef object subclass
+        cdef OBIDMS_column_p column_p
+        
+        column_name_b = str2bytes(column_name)
+        
+        if nb_elements_per_line > 1 :
+            elements_names_b = str2bytes(';'.join(elements_names))
+        elif nb_elements_per_line == 1 :
+            elements_names_b = column_name_b
+    
+        if type :    
+            if type == 'OBI_INT' :
+                data_type = OBI_INT
+            elif type == 'OBI_FLOAT' :
+                data_type = OBI_FLOAT
+            elif type == 'OBI_BOOL' :
+                data_type = OBI_BOOL
+            elif type == 'OBI_CHAR' :
+                data_type = OBI_CHAR
+            elif type == 'OBI_QUAL' :
+                data_type = OBI_QUAL
+            elif type == 'OBI_STR' :
+                data_type = OBI_STR
+            elif type == 'OBI_SEQ' :
+                data_type = OBI_SEQ
+            else :
+                raise Exception("Invalid provided data type")
+        
+        if (obi_view_add_column(self.pointer, column_name_b, version_number,     # TODO should return pointer on column?
+                                data_type, nb_lines, nb_elements_per_line, 
+                                elements_names_b, str2bytes(indexer_name), 
+                                str2bytes(comments), create) < 0) :
+            raise Exception("Problem adding a column in a view")
+        
+        # Get the column pointer
+        column_p = obi_view_get_column(self.pointer, column_name_b)
+        
+        # Open and store the subclass
+        subclass = OBIDMS_column.get_subclass_type(column_p)
+        (self.columns)[column_name] = subclass(self, column_name)
+                
+    
+    cpdef save_and_close(self) :
+        if (obi_save_and_close_view(self.pointer) < 0) :
+            raise Exception("Problem closing a view")
+    
+
+    def __iter__(self):
+        # iter on each line of all columns
+        
+        # Declarations
+        cdef index_t      lines_used
+        cdef index_t      line_nb
+        cdef OBIView_line line          # TODO Check that this works for NUC SEQ views
+        
+        # Yield each line
+        lines_used = self.pointer.infos.line_count
+        
+        for line_nb in range(lines_used) :
+            line = self[line_nb]
+            yield line
+
+
+    def __getitem__(self, object item) :
+        if type(item) == str :
+            return (self.columns)[item]
+        elif type(item) == int :
+            return OBIView_line(self, item)
+
+
+    cpdef select_line(self, index_t line_nb) :
+        if obi_select_line(self.pointer, line_nb) < 0 :
+            raise Exception("Problem selecting a line")
+
+    
+    cpdef select_lines(self, list line_selection) :
+        cdef index_t* line_selection_p
+        line_selection_p = <index_t*> malloc((len(line_selection) + 1) * sizeof(index_t))
+        for i in range(len(line_selection)) :
+            line_selection_p[i] = line_selection[i]
+        line_selection_p[len(line_selection)] = -1
+        if obi_select_lines(self.pointer, line_selection_p) < 0 :
+            raise Exception("Problem selecting a list of lines")
+    
+    
+    def __contains__(self, str column_name):
+        return (column_name in self.columns)
+    
+    
+    def __str__(self) :
+        cdef OBIView_line line
+        cdef str to_print
+        to_print = ""
+        for line in self.__iter__() :
+            to_print = to_print + str(line) + "\n"
+        return to_print
+
+#############################################
+
+cdef class OBIView_NUC_SEQS(OBIView):
+    
+    def __init__(self, OBIDMS dms, str view_name, bint new=False, object view_to_clone=None, list line_selection=None, str comments=""): 
+        
+        cdef Obiview_p              view = NULL
+        cdef int                    i
+        cdef list                   col_list
+        cdef str                    col_name
+        cdef OBIDMS_column          column
+        cdef OBIDMS_column_p        column_p
+        cdef OBIDMS_column_header_p header
+        cdef index_t*               line_selection_p
+                
+        self.dms = dms
+        
+        if line_selection is not None :
+            line_selection_p = <index_t*> malloc((len(line_selection) + 1) * sizeof(index_t))
+            for i in range(len(line_selection)) :
+                line_selection_p[i] = line_selection[i]
+            line_selection_p[len(line_selection)] = -1
+        else :
+            line_selection_p = NULL
+       
+        if new :
+            if view_to_clone is not None :
+                if type(view_to_clone) == str :
+                    view = obi_new_view_nuc_seqs_cloned_from_name(dms.pointer, str2bytes(view_name), str2bytes(view_to_clone), line_selection_p, str2bytes(comments))
+                else :
+                    view = obi_new_view_nuc_seqs(dms.pointer, str2bytes(view_name), (<OBIView> view_to_clone).pointer, line_selection_p, str2bytes(comments))
+            elif view_to_clone is None :
+                view = obi_new_view_nuc_seqs(dms.pointer, str2bytes(view_name), NULL, line_selection_p, str2bytes(comments))
+        elif not new :
+            if view_name is not None :
+                view = obi_open_view(dms.pointer, str2bytes(view_name))
+            elif view_name is None :
+                view = obi_open_view(dms.pointer, NULL)
+        
+        if view == NULL :
+            raise Exception("Error creating/opening view")
+
+        self.pointer = view
+        self.name = bytes2str(view.infos.name)
+        self.comments = bytes2str(view.infos.comments)
+        
+        # Go through columns to build list of corresponding python instances
+        self.columns = {}
+        for i in range(view.infos.column_count) :
+            column_p               = <OBIDMS_column_p> (view.columns)[i]
+            header                 = (column_p).header
+            col_name               = bytes2str(header.name)         
+            subclass               = OBIDMS_column.get_subclass_type(column_p)
+            self.columns[col_name] = subclass(self, col_name)
+        
+        self.ids = self.columns[bytes2str(ID_COLUMN)]
+        self.sequences = self.columns[bytes2str(NUC_SEQUENCE_COLUMN)]
+        self.definitions = self.columns[bytes2str(DEFINITION_COLUMN)]
+        self.qualities = self.columns[bytes2str(QUALITY_COLUMN)]
+            
+    def __getitem__(self, object item) :
+        if type(item) == str :
+            return (self.columns)[item]
+        elif type(item) == int :
+            return OBI_Nuc_Seq_Stored(self, item)
+    
+    def __setitem__(self, index_t line_idx, OBI_Nuc_Seq sequence_obj) :
+        for key in sequence_obj :
+            self[line_idx][key] = sequence_obj[key]
+            
+
+#############################################
+
+cdef class OBIView_line :
+
+    def __init__(self, OBIView view, index_t line_nb) :
+        self.index = line_nb
+        self.view = view
+    
+    def __getitem__(self, str column_name) :
+        return ((self.view).columns)[column_name][self.index]
+    
+    def __setitem__(self, str column_name, object value):
+        # TODO detect multiple elements (dict type)? put somewhere else? but more risky (in get)
+        # TODO OBI_QUAL ?
+        cdef type value_type
+        cdef str value_obitype
+        if column_name not in self.view :
+            if value == None :
+                raise Exception("Trying to create a column from a None value (can't guess type)")
+            value_type = type(value)
+            if value_type == int :
+                value_obitype = 'OBI_INT'
+            elif value_type == float :
+                value_obitype = 'OBI_FLOAT'
+            elif value_type == bool :
+                value_obitype = 'OBI_BOOL'            
+            elif value_type == str :
+                if only_ATGC(str2bytes(value)) :    # TODO detect IUPAC?
+                    value_obitype = 'OBI_SEQ'
+                elif len(value) == 1 :
+                    value_obitype = 'OBI_CHAR'
+                elif (len(value) > 1) :
+                    value_obitype = 'OBI_STR'
+            else :
+                raise Exception("Could not guess the type of a value to create a new column")
+            self.view.add_column(column_name, type=value_obitype)
+
+        (((self.view).columns)[column_name]).set_line(self.index, value)
+
+    def __contains__(self, str column_name):
+        return (column_name in self.view)
+        
+    def __repr__(self):
+        cdef dict line
+        cdef str column_name
+        line = {}
+        for column_name in self.view.columns :
+            line[column_name] = self[column_name]
+        return str(line)
+
+##########################################
 
 cdef class OBIDMS :
     
@@ -49,235 +578,127 @@ cdef class OBIDMS :
         # Fill structure and create or open the DMS
         self.dms_name = dms_name
         self.pointer = obi_dms(<const_char_p> dms_name_b)
+        if self.pointer == NULL :
+            raise Exception("Failed opening or creating an OBIDMS")
 
 
-#    def __del__(self) :                        # TODO problem with closing dir breaking everything
-#        obi_close_dms(self.pointer)
+    cpdef close(self) :
+        if (obi_close_dms(self.pointer)) < 0 :
+            raise Exception("Problem closing an OBIDMS")
 
     
-    cpdef dict list(self):
+    cpdef OBI_Taxonomy open_taxonomy(self, str taxo_name) :
+        return OBI_Taxonomy(self, taxo_name)
     
-        # Declarations
-        cdef object p
-        cdef dict dms = {}
-        cdef str column_name
-        cdef bytes column_name_b
-        cdef str data_type
-        cdef obiversion_t latest_version
-        cdef size_t line_count
 
-        p = Path(self.dms_name+'.obidms')
+    cpdef OBIView open_view(self, str view_name) :
         
-        print("{:<25} {:<25} {:<25} {:<25}".format('-Column name-','-Data type-','-Latest version number-', '-Line count of latest version-'))
-        for entry in p.iterdir():
-            if entry.suffix == ".obicol":
-                column_name = entry.stem
-                column_name_b = str2bytes(column_name)
-                dms[column_name] = {}
-                data_type = bytes2str(name_data_type(obi_column_get_data_type_from_name(self.pointer, column_name_b)))
-                latest_version = obi_column_get_latest_version_from_name(self.pointer, column_name_b)
-                line_count = obi_column_get_line_count_from_name(self.pointer, column_name_b)
-                dms[column_name]['data_type'] = data_type
-                dms[column_name]['latest_version'] = latest_version
-                dms[column_name]['line_count'] = line_count
-                print("{:<25} {:<25} {:<25} {:<25}".format(column_name, data_type, latest_version, line_count))
+        cdef object view_class
+        cdef dict view_infos
         
-        return dms
+        view_infos = self.read_view_infos(view_name)
         
-    cpdef OBIDMS_column open_column(self,
-                                    str column_name,
-                                    bint create=False, 
-                                    bint clone=False, bint clone_data=True,
-                                    obiversion_t version_number=-1, 
-                                    OBIType_t data_type= <OBIType_t> 0,
-                                    size_t nb_lines=0, 
-                                    size_t nb_elements_per_line=1, 
-                                    str elements_names=None):
+        if view_infos["view_type"] == bytes2str(VIEW_TYPE_NUC_SEQS) :
+            view_class = OBIView_NUC_SEQS
+        else :
+            view_class = OBIView
                 
-        # Declarations
-        cdef OBIDMS_column column
-        cdef bytes column_name_b
+        return view_class(self, view_name)
     
-        # Format the character string to send to C function
-        column_name_b = str2bytes(column_name)
 
-        # Get the data type if not provided
-        if not data_type :
-            if create :
-                raise Exception("A data type must be specified")
-            else :
-                data_type = obi_column_get_data_type_from_name(self.pointer, column_name_b)
+    cpdef OBIView new_view(self, str view_name, object view_to_clone=None, list line_selection=None, str view_type=None, str comments="") :
         
-        # Open the column with the right subclass depending on the data type and the mode (read-only or writable)
-        if data_type == 1 :
-            if (create or clone) :
-                column = OBIDMS_column_int_writable(self, column_name, 
-                                                    create, clone, clone_data, 
-                                                    version_number, data_type, 
-                                                    nb_lines, nb_elements_per_line, 
-                                                    elements_names)
-            else :
-                 column = OBIDMS_column_int(self, column_name, 
-                                                 create, clone, clone_data, 
-                                                 version_number, data_type, 
-                                                 nb_lines, nb_elements_per_line, 
-                                                 elements_names)
+        cdef object view_class
         
-        elif data_type == 2 :
-            if (create or clone) :
-                column = OBIDMS_column_float_writable(self, column_name, 
-                                                      create, clone, clone_data, 
-                                                      version_number, data_type, 
-                                                      nb_lines, nb_elements_per_line, 
-                                                      elements_names)
-            else :
-                 column = OBIDMS_column_float(self, column_name, 
-                                                   create, clone, clone_data, 
-                                                   version_number, data_type, 
-                                                   nb_lines, nb_elements_per_line, 
-                                                   elements_names)
+        if view_type is not None :
+            if view_type == bytes2str(VIEW_TYPE_NUC_SEQS) :
+                view_class = OBIView_NUC_SEQS
+        else :
+            view_class = OBIView
         
-        elif data_type == 3 :
-            if (create or clone) :
-                column = OBIDMS_column_bool_writable(self, column_name, 
-                                                     create, clone, clone_data, 
-                                                     version_number, data_type, 
-                                                     nb_lines, nb_elements_per_line, 
-                                                     elements_names)
-            else :
-                 column = OBIDMS_column_bool(self, column_name, 
-                                                  create, clone, clone_data, 
-                                                  version_number, data_type, 
-                                                  nb_lines, nb_elements_per_line, 
-                                                  elements_names)
+        return view_class(self, view_name, new=True, view_to_clone=view_to_clone, line_selection=line_selection, comments=comments)
 
-        elif data_type == 4 :
-            if (create or clone) :
-                column = OBIDMS_column_char_writable(self, column_name, 
-                                                     create, clone, clone_data, 
-                                                     version_number, data_type, 
-                                                     nb_lines, nb_elements_per_line, 
-                                                     elements_names)
-            else :
-                 column = OBIDMS_column_char(self, column_name, 
-                                                  create, clone, clone_data, 
-                                                  version_number, data_type, 
-                                                  nb_lines, nb_elements_per_line, 
-                                                  elements_names)
 
-#         elif data_type == 5 :
-#             if (create or clone) :
-#                 column = OBIDMS_column_idx_writable(self, column_name, 
-#                                                     create, clone, clone_data, 
-#                                                     version_number, data_type, 
-#                                                     nb_lines, nb_elements_per_line, 
-#                                                     elements_names)
+    cpdef dict read_view_infos(self, str view_name) :
+        
+        cdef Obiview_infos_p      view_infos_p
+        cdef dict                 view_infos_d
+        cdef Column_reference_p   column_refs
+        cdef int                  i, j
+        cdef str                  column_name
+
+        view_infos_p = obi_view_map_file(self.pointer, str2bytes(view_name))
+        view_infos_d = {}
+        view_infos_d["name"] = bytes2str(view_infos_p.name)
+        view_infos_d["comments"] = bytes2str(view_infos_p.comments)
+        view_infos_d["view_type"] = bytes2str(view_infos_p.view_type)
+        view_infos_d["column_count"] = <int> view_infos_p.column_count
+        view_infos_d["line_count"] = <int> view_infos_p.line_count
+        view_infos_d["created_from"] = bytes2str(view_infos_p.created_from)
+        view_infos_d["creation_date"] = bytes2str(obi_format_date(view_infos_p.creation_date))
+        if (view_infos_p.all_lines) :
+            view_infos_d["line_selection"] = None
+        else :
+            view_infos_d["line_selection"] = {}
+            view_infos_d["line_selection"]["column_name"] = bytes2str((view_infos_p.line_selection).column_name)
+            view_infos_d["line_selection"]["version"] = <int> (view_infos_p.line_selection).version
+        view_infos_d["column_references"] = {}
+        column_refs = view_infos_p.column_references
+        for j in range(view_infos_d["column_count"]) :
+            column_name = bytes2str((column_refs[j]).column_name)
+            view_infos_d["column_references"][column_name] = {}
+            view_infos_d["column_references"][column_name]["version"] = column_refs[j].version
+        
+        obi_view_unmap_file(self.pointer, view_infos_p)
+
+        return view_infos_d
+
+
+#     cpdef dict read_views(self) :    # TODO function that prints the dic nicely and function that prints 1 view nicely. Add column type in col ref
+#         
+#         cdef Obiviews_infos_all_p all_views_p
+#         cdef Obiview_infos_p      view_p
+#         cdef Column_reference_p   column_refs
+#         cdef int                  nb_views
+#         cdef int                  i, j
+#         cdef str                  view_name
+#         cdef str                  column_name
+#         cdef dict                 views
+#         cdef bytes                name_b
+#         
+#         views = {}
+#         all_views_p = obi_read_view_infos(self.pointer)
+#         if all_views_p == NULL :
+#             raise Exception("No views to read")
+#         nb_views = <int> (all_views_p.header).view_count
+#         for i in range(nb_views) :
+#             view_p = (<Obiview_infos_p> (all_views_p.view_infos)) + i
+#             view_name = bytes2str(view_p.name)
+#             views[view_name] = {}
+#             views[view_name]["comments"] = bytes2str(view_p.comments)
+#             views[view_name]["view_type"] = bytes2str(view_p.view_type)
+#             views[view_name]["column_count"] = <int> view_p.column_count
+#             views[view_name]["line_count"] = <int> view_p.line_count
+#             views[view_name]["view_number"] = <int> view_p.view_number
+#             views[view_name]["created_from"] = bytes2str(view_p.created_from)
+#             views[view_name]["creation_date"] = bytes2str(obi_format_date(view_p.creation_date))
+#             if (view_p.all_lines) :
+#                 views[view_name]["line_selection"] = None
 #             else :
-#                  column = OBIDMS_column_idx(self, column_name, 
-#                                             create, clone, clone_data, 
-#                                             version_number, data_type, 
-#                                             nb_lines, nb_elements_per_line, 
-#                                             elements_names)
-   
-        else :
-            raise Exception("Problem with the data type")
-            
-        return column
+#                 views[view_name]["line_selection"] = {}
+#                 views[view_name]["line_selection"]["column_name"] = bytes2str((view_p.line_selection).column_name)
+#                 views[view_name]["line_selection"]["version"] = <int> (view_p.line_selection).version
+#             views[view_name]["column_references"] = {}
+#             column_refs = view_p.column_references
+#             for j in range(views[view_name]["column_count"]) :
+#                 column_name = bytes2str((column_refs[j]).column_name)
+#                 views[view_name]["column_references"][column_name] = {}
+#                 views[view_name]["column_references"][column_name]["version"] = column_refs[j].version
+#         
+#         obi_close_view_infos(all_views_p);
+#         
+#         return views
     
     
     
-cdef class OBIDMS_column :
-
-    # Should only be initialized through a subclass
-    def __init__(self, 
-                 OBIDMS dms, 
-                 str column_name,
-                 bint create, 
-                 bint clone, bint clone_data,
-                 obiversion_t version_number, 
-                 OBIType_t type,
-                 size_t nb_lines, 
-                 size_t nb_elements_per_line, 
-                 str elements_names):
-        
-        # Declarations
-        cdef bytes column_name_b
-        cdef bytes dms_name_b
-        cdef bytes elements_names_b
-        
-        # Fill structure
-        self.dms = dms
-        self.data_type = bytes2str(name_data_type(type))
-        self.column_name = column_name
-        
-        # Format the character strings to send them to C functions
-        column_name_b = str2bytes(column_name)
-        dms_name_b = str2bytes(self.dms.dms_name)
-        
-        # Create, clone or open column
-        if create :
-            if elements_names == None :  
-                elements_names_b = column_name_b
-            else :
-                elements_names_b = str2bytes(elements_names)
-            self.pointer = obi_create_column(self.dms.pointer, column_name_b, type, nb_lines, nb_elements_per_line, elements_names_b)
-        else :
-            if clone :
-                self.pointer = obi_clone_column(self.dms.pointer, column_name_b, version_number, clone_data)
-            else :
-                self.pointer = obi_open_column(self.dms.pointer, column_name_b, version_number)
-    
-    
-    def __iter__(self):
-    
-        # Declarations
-        cdef list elements_names
-        cdef str element_name
-        cdef bint multiple_elements
-        cdef object line               # TODO
-        cdef size_t lines_used
-        cdef size_t line_nb
-        
-        # Check if there are multiple elements per line anf if yes, get their names
-        elements_names = self.get_elements_names()
-        if len(elements_names) > 1 :
-            multiple_elements = True
-        else :
-            element_name = elements_names[0]
-        
-        # Yield each line
-        lines_used = obi_column_get_nb_lines_used(self.pointer)
-        for line_nb in xrange(lines_used):
-            if multiple_elements :
-                line = []
-                for element_name in elements_names :
-                    line.append(self.get_item(line_nb, element_name))
-            else :
-                line = self.get_item(line_nb, element_name)
-            yield line
-
-
-    def __setitem__(self, size_t line_nb, object value):
-        self.set_item(line_nb, "", value)
-
-        
-    def __getitem__(self, size_t line_nb):
-        return self.get_item(line_nb, "")
-
-    cpdef object get_item(self, size_t line_nb, str element_name):
-        raise NotImplementedError
-    
-    cpdef list get_elements_names(self):
-        cdef bytes elements_names
-        elements_names = obi_column_get_elements_names(self.pointer)
-        return (bytes2str(elements_names)).split(';')
-
-             
-    cpdef str get_data_type(self):
-        return self.data_type
-
-    
-    cpdef size_t get_nb_lines_used(self):
-        return obi_column_get_nb_lines_used(self.pointer)
-        
     

python/obitools3/obidms/_obidmscolumn_bool.cfiles

@@ -1,16 +1,59 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
 ../../../src/obidmscolumn_bool.c
 ../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
 ../../../src/obidmscolumndir.c
-../../../src/obidms.h
-../../../src/obidms.c
 ../../../src/obierrno.h
 ../../../src/obierrno.c
 ../../../src/obilittlebigman.h
 ../../../src/obilittlebigman.c
 ../../../src/obitypes.h
 ../../../src/obitypes.c
-../../../src/private_at_functions.h
-../../../src/private_at_functions.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
+../../../src/utils.h
+../../../src/utils.c

python/obitools3/obidms/_obidmscolumn_bool.pxd

@@ -1,17 +1,14 @@
 #cython: language_level=3
 
-from .capi.obitypes cimport obibool_t
-from .capi.obidmscolumn cimport OBIDMS_column_p
-                                                
-from ._obidms cimport OBIDMS_column
+from .capi.obitypes cimport index_t
+from ._obidms       cimport OBIDMS_column, OBIDMS_column_multi_elts
 
 
 cdef class OBIDMS_column_bool(OBIDMS_column):
-    cpdef object get_item(self, size_t line_nb, str element_name)
-    cpdef set_item(self, size_t line_nb, str element_name, obibool_t value)
-    cpdef close(self)
-
-cdef class OBIDMS_column_bool_writable(OBIDMS_column_bool):
-    cpdef set_item(self, size_t line_nb, str element_name, obibool_t value)
-    cpdef close(self)
+    cpdef object get_line(self, index_t line_nb)
+    cpdef set_line(self, index_t line_nb, object value)
 
+cdef class OBIDMS_column_multi_elts_bool(OBIDMS_column_multi_elts):
+    cpdef object get_item(self, index_t line_nb, str element_name)
+    cpdef object get_line(self, index_t line_nb)
+    cpdef set_item(self, index_t line_nb, str element_name, object value)

python/obitools3/obidms/_obidmscolumn_bool.pyx

@@ -1,11 +1,11 @@
 #cython: language_level=3
  
-from .capi.obidmscolumn cimport obi_close_column,\
-                                obi_truncate_and_close_column, \
-                                obi_column_get_obibool_with_elt_name, \
-                                obi_column_set_obibool_with_elt_name
+from .capi.obiview      cimport obi_column_get_obibool_with_elt_name_in_view, \
+                                obi_column_get_obibool_with_elt_idx_in_view, \
+                                obi_column_set_obibool_with_elt_name_in_view, \
+                                obi_column_set_obibool_with_elt_idx_in_view
 from .capi.obierrno     cimport obi_errno
-from .capi.obitypes     cimport OBIBool_NA
+from .capi.obitypes     cimport OBIBool_NA, obibool_t
 
 from obitools3.utils cimport str2bytes
 
@@ -14,10 +14,31 @@ from cpython.bool cimport PyBool_FromLong
 
 cdef class OBIDMS_column_bool(OBIDMS_column):
 
-    cpdef object get_item(self, size_t line_nb, str element_name):
+    cpdef object get_line(self, index_t line_nb):
         cdef obibool_t value
         cdef object result
-        value = obi_column_get_obibool_with_elt_name(self.pointer, line_nb, str2bytes(element_name))
+        value = obi_column_get_obibool_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0)
+        if obi_errno > 0 :
+            raise IndexError(line_nb)
+        if value == OBIBool_NA :
+            result = None
+        else :
+            result = PyBool_FromLong(value)
+        return result
+    
+    cpdef set_line(self, index_t line_nb, object value):
+        if value is None :
+            value = OBIBool_NA
+        if obi_column_set_obibool_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, <obibool_t> value) < 0:
+            raise Exception("Problem setting a value in a column")
+
+
+cdef class OBIDMS_column_multi_elts_bool(OBIDMS_column_multi_elts):
+
+    cpdef object get_item(self, index_t line_nb, str element_name):
+        cdef obibool_t value
+        cdef object result
+        value = obi_column_get_obibool_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name))
         if obi_errno > 0 :
             raise IndexError(line_nb, element_name)
         if value == OBIBool_NA :
@@ -26,21 +47,31 @@ cdef class OBIDMS_column_bool(OBIDMS_column):
             result = PyBool_FromLong(value)
         return result
     
-    cpdef set_item(self, size_t line_nb, str element_name, obibool_t value):
-        raise Exception("Column is read-only")
+    cpdef object get_line(self, index_t line_nb) :
+        cdef obibool_t value
+        cdef object value_in_result
+        cdef dict result
+        cdef index_t i
+        cdef bint all_NA
+        result = {}
+        all_NA = True
+        for i in range(self.nb_elements_per_line) :
+            value = obi_column_get_obibool_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, i)
+            if obi_errno > 0 :
+                raise IndexError(line_nb)
+            if value == OBIBool_NA :
+                value_in_result = None
+            else :
+                value_in_result = PyBool_FromLong(value)   
+            result[self.elements_names[i]] = value_in_result
+            if all_NA and (value_in_result is not None) :
+                all_NA = False
+        if all_NA :
+            result = None
+        return result
 
-    cpdef close(self):
-        if obi_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
-
-
-cdef class OBIDMS_column_bool_writable(OBIDMS_column_bool):     
-   
-    cpdef set_item(self, size_t line_nb, str element_name, obibool_t value):
-        if obi_column_set_obibool_with_elt_name(self.pointer, line_nb, str2bytes(element_name), value) < 0 :
+    cpdef set_item(self, index_t line_nb, str element_name, object value):
+        if value is None :
+            value = OBIBool_NA
+        if obi_column_set_obibool_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), <obibool_t> value) < 0:
             raise Exception("Problem setting a value in a column")
- 
-    cpdef close(self):
-        if obi_truncate_and_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
-     

python/obitools3/obidms/_obidmscolumn_char.cfiles

@@ -1,16 +1,59 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
 ../../../src/obidmscolumn_char.c
 ../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
 ../../../src/obidmscolumndir.c
-../../../src/obidms.h
-../../../src/obidms.c
 ../../../src/obierrno.h
 ../../../src/obierrno.c
 ../../../src/obilittlebigman.h
 ../../../src/obilittlebigman.c
 ../../../src/obitypes.h
 ../../../src/obitypes.c
-../../../src/private_at_functions.h
-../../../src/private_at_functions.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
+../../../src/utils.h
+../../../src/utils.c

python/obitools3/obidms/_obidmscolumn_char.pxd

@@ -1,17 +1,14 @@
 #cython: language_level=3
 
-from .capi.obitypes cimport obichar_t
-from .capi.obidmscolumn cimport OBIDMS_column_p
-                                                
-from ._obidms cimport OBIDMS_column
+from .capi.obitypes cimport index_t
+from ._obidms       cimport OBIDMS_column, OBIDMS_column_multi_elts
 
 
 cdef class OBIDMS_column_char(OBIDMS_column):
-    cpdef object get_item(self, size_t line_nb, str element_name)
-    cpdef set_item(self, size_t line_nb, str element_name, bytes value)
-    cpdef close(self)
-
-cdef class OBIDMS_column_char_writable(OBIDMS_column_char):
-    cpdef set_item(self, size_t line_nb, str element_name, bytes value)
-    cpdef close(self)
+    cpdef object get_line(self, index_t line_nb)
+    cpdef set_line(self, index_t line_nb, object value)
 
+cdef class OBIDMS_column_multi_elts_char(OBIDMS_column_multi_elts):
+    cpdef object get_item(self, index_t line_nb, str element_name)
+    cpdef object get_line(self, index_t line_nb)
+    cpdef set_item(self, index_t line_nb, str element_name, object value)

python/obitools3/obidms/_obidmscolumn_char.pyx

@@ -1,44 +1,76 @@
 #cython: language_level=3
  
-from .capi.obidmscolumn cimport obi_close_column,\
-                                obi_truncate_and_close_column, \
-                                obi_column_get_obichar_with_elt_name, \
-                                obi_column_set_obichar_with_elt_name
+from .capi.obiview      cimport obi_column_get_obichar_with_elt_name_in_view, \
+                                obi_column_get_obichar_with_elt_idx_in_view, \
+                                obi_column_set_obichar_with_elt_name_in_view, \
+                                obi_column_set_obichar_with_elt_idx_in_view
 from .capi.obierrno     cimport obi_errno
-from .capi.obitypes     cimport OBIChar_NA
+from .capi.obitypes     cimport OBIChar_NA, obichar_t
 
-from obitools3.utils cimport str2bytes
+from obitools3.utils cimport str2bytes, bytes2str
 
 
-cdef class OBIDMS_column_char(OBIDMS_column) :
+cdef class OBIDMS_column_char(OBIDMS_column):
 
-    cpdef object get_item(self, size_t line_nb, str element_name):
-        cdef char value
+    cpdef object get_line(self, index_t line_nb):
+        cdef obichar_t value
         cdef object result
-        value = obi_column_get_obichar_with_elt_name(self.pointer, line_nb, str2bytes(element_name))
+        value = obi_column_get_obichar_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0)
+        if obi_errno > 0 :
+            raise IndexError(line_nb)
+        if value == OBIChar_NA :
+            result = None
+        else :
+            result = bytes2str(value)
+        return result
+
+    cpdef set_line(self, index_t line_nb, object value):
+        if value is None :
+            value = OBIChar_NA
+        if obi_column_set_obichar_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, str2bytes(value)[0]) < 0:
+            raise Exception("Problem setting a value in a column")
+
+
+cdef class OBIDMS_column_multi_elts_char(OBIDMS_column_multi_elts):
+
+    cpdef object get_item(self, index_t line_nb, str element_name):
+        cdef obichar_t value
+        cdef object result
+        value = obi_column_get_obichar_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name))
         if obi_errno > 0 :
             raise IndexError(line_nb, element_name)
         if value == OBIChar_NA :
             result = None
         else :
-            result = <bytes> value
+            result = bytes2str(value)
         return result
 
-    cpdef set_item(self, size_t line_nb, str element_name, bytes value):
-        raise Exception("Column is read-only")
+    cpdef object get_line(self, index_t line_nb) :
+        cdef obichar_t value
+        cdef object value_in_result
+        cdef dict result
+        cdef index_t i
+        cdef bint all_NA
+        result = {}
+        all_NA = True
+        for i in range(self.nb_elements_per_line) :
+            value = obi_column_get_obichar_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, i)
+            if obi_errno > 0 :
+                raise IndexError(line_nb)
+            if value == OBIChar_NA :
+                value_in_result = None
+            else :
+                value_in_result = bytes2str(value)   
+            result[self.elements_names[i]] = value_in_result
+            if all_NA and (value_in_result is not None) :
+                all_NA = False
+        if all_NA :
+            result = None
+        return result
 
-    cpdef close(self):
-        if obi_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
- 
-         
-cdef class OBIDMS_column_char_writable(OBIDMS_column_char) :     
- 
-    cpdef set_item(self, size_t line_nb, str element_name, bytes value):
-        if obi_column_set_obichar_with_elt_name(self.pointer, line_nb, str2bytes(element_name), value[0]) < 0:
+    cpdef set_item(self, index_t line_nb, str element_name, object value):
+        if value is None :
+            value = OBIChar_NA
+        if obi_column_set_obichar_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), str2bytes(value)[0]) < 0:
             raise Exception("Problem setting a value in a column")
 
-    cpdef close(self):
-        if obi_truncate_and_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
-      

python/obitools3/obidms/_obidmscolumn_float.cfiles

@@ -1,16 +1,59 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
 ../../../src/obidmscolumn_float.c
 ../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
 ../../../src/obidmscolumndir.c
-../../../src/obidms.h
-../../../src/obidms.c
 ../../../src/obierrno.h
 ../../../src/obierrno.c
 ../../../src/obilittlebigman.h
 ../../../src/obilittlebigman.c
 ../../../src/obitypes.h
 ../../../src/obitypes.c
-../../../src/private_at_functions.h
-../../../src/private_at_functions.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
+../../../src/utils.h
+../../../src/utils.c

python/obitools3/obidms/_obidmscolumn_float.pxd

@@ -1,16 +1,14 @@
 #cython: language_level=3
 
-from .capi.obitypes cimport obifloat_t
-from .capi.obidmscolumn cimport OBIDMS_column_p
-                                                
-from ._obidms cimport OBIDMS_column
+from .capi.obitypes cimport index_t
+from ._obidms       cimport OBIDMS_column, OBIDMS_column_multi_elts
 
 
 cdef class OBIDMS_column_float(OBIDMS_column):
-    cpdef object get_item(self, size_t line_nb, str element_name)
-    cpdef set_item(self, size_t line_nb, str element_name, obifloat_t value)
-    cpdef close(self)
+    cpdef object get_line(self, index_t line_nb)
+    cpdef set_line(self, index_t line_nb, object value)
 
-cdef class OBIDMS_column_float_writable(OBIDMS_column_float):
-    cpdef set_item(self, size_t line_nb, str element_name, obifloat_t value)
-    cpdef close(self)
+cdef class OBIDMS_column_multi_elts_float(OBIDMS_column_multi_elts):
+    cpdef object get_item(self, index_t line_nb, str element_name)
+    cpdef object get_line(self, index_t line_nb)
+    cpdef set_item(self, index_t line_nb, str element_name, object value)

python/obitools3/obidms/_obidmscolumn_float.pyx

@@ -1,44 +1,76 @@
 #cython: language_level=3
 
-from .capi.obidmscolumn cimport obi_close_column,\
-                                obi_truncate_and_close_column, \
-                                obi_column_get_obifloat_with_elt_name, \
-                                obi_column_set_obifloat_with_elt_name
+from .capi.obiview      cimport obi_column_get_obifloat_with_elt_name_in_view, \
+                                obi_column_get_obifloat_with_elt_idx_in_view, \
+                                obi_column_set_obifloat_with_elt_name_in_view, \
+                                obi_column_set_obifloat_with_elt_idx_in_view
 from .capi.obierrno     cimport obi_errno
-from .capi.obitypes     cimport OBIFloat_NA
+from .capi.obitypes     cimport OBIFloat_NA, obifloat_t
 
 from obitools3.utils cimport str2bytes
 
 
 cdef class OBIDMS_column_float(OBIDMS_column):
 
-    cpdef object get_item(self, size_t line_nb, str element_name):
+    cpdef object get_line(self, index_t line_nb):
         cdef obifloat_t value
         cdef object result
-        value = obi_column_get_obifloat_with_elt_name(self.pointer, line_nb, str2bytes(element_name))
+        value = obi_column_get_obifloat_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0)
+        if obi_errno > 0 :
+            raise IndexError(line_nb)
+        if value == OBIFloat_NA :
+            result = None
+        else :
+            result = <double> value
+        return result
+
+    cpdef set_line(self, index_t line_nb, object value):
+        if value is None :
+            value = OBIFloat_NA
+        if obi_column_set_obifloat_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, <obifloat_t> value) < 0:
+            raise Exception("Problem setting a value in a column")
+
+
+cdef class OBIDMS_column_multi_elts_float(OBIDMS_column_multi_elts):
+
+    cpdef object get_item(self, index_t line_nb, str element_name):
+        cdef obifloat_t value
+        cdef object result
+        value = obi_column_get_obifloat_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name))
         if obi_errno > 0 :
             raise IndexError(line_nb, element_name)
         if value == OBIFloat_NA :
             result = None
         else :
-            result = <float> value
+            result = <double> value
         return result
 
-    cpdef set_item(self, size_t line_nb, str element_name, obifloat_t value):
-        raise Exception("Column is read-only")
+    cpdef object get_line(self, index_t line_nb) :
+        cdef obifloat_t value
+        cdef object value_in_result
+        cdef dict result
+        cdef index_t i
+        cdef bint all_NA
+        result = {}
+        all_NA = True
+        for i in range(self.nb_elements_per_line) :
+            value = obi_column_get_obifloat_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, i)
+            if obi_errno > 0 :
+                raise IndexError(line_nb)
+            if value == OBIFloat_NA :
+                value_in_result = None
+            else :
+                value_in_result = <double> value
+            result[self.elements_names[i]] = value_in_result
+            if all_NA and (value_in_result is not None) :
+                all_NA = False
+        if all_NA :
+            result = None
+        return result
    
-    cpdef close(self):
-        if obi_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
-                  
-
-cdef class OBIDMS_column_float_writable(OBIDMS_column_float):     
-
-    cpdef set_item(self, size_t line_nb, str element_name, obifloat_t value):
-        if obi_column_set_obifloat_with_elt_name(self.pointer, line_nb, str2bytes(element_name), value) < 0:
+    cpdef set_item(self, index_t line_nb, str element_name, object value):
+        if value is None :
+            value = OBIFloat_NA
+        if obi_column_set_obifloat_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), <obifloat_t> value) < 0:
             raise Exception("Problem setting a value in a column")
  
-    cpdef close(self):
-        if obi_truncate_and_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
- 

python/obitools3/obidms/_obidmscolumn_idx.cfiles

@@ -1,16 +0,0 @@
-../../../src/obidmscolumn_idx.c
-../../../src/obidmscolumn_idx.h
-../../../src/obidmscolumn.h
-../../../src/obidmscolumn.c
-../../../src/obidmscolumndir.h
-../../../src/obidmscolumndir.c
-../../../src/obidms.h
-../../../src/obidms.c
-../../../src/obierrno.h
-../../../src/obierrno.c
-../../../src/obilittlebigman.h
-../../../src/obilittlebigman.c
-../../../src/obitypes.h
-../../../src/obitypes.c
-../../../src/private_at_functions.h
-../../../src/private_at_functions.c

python/obitools3/obidms/_obidmscolumn_idx.pxd

@@ -1,16 +0,0 @@
-#cython: language_level=3
-
-from .capi.obitypes cimport obiidx_t
-from .capi.obidmscolumn cimport OBIDMS_column_p
-                                                
-from ._obidms cimport OBIDMS_column
-
-
-cdef class OBIDMS_column_idx(OBIDMS_column):
-    cpdef object get_item(self, size_t line_nb, str element_name)
-    cpdef set_item(self, size_t line_nb, str element_name, obiidx_t value)
-    cpdef close(self)
-    
-cdef class OBIDMS_column_idx_writable(OBIDMS_column_idx):
-    cpdef set_item(self, size_t line_nb, str element_name, obiidx_t value)
-    cpdef close(self)

python/obitools3/obidms/_obidmscolumn_idx.pyx

@@ -1,46 +0,0 @@
-#cython: language_level=3
-
-from .capi.obidmscolumn cimport obi_close_column,\
-                                obi_truncate_and_close_column, \
-                                obi_column_get_obiidx_with_elt_name, \
-                                obi_column_set_obiidx_with_elt_name
-from .capi.obierrno     cimport obi_errno
-from .capi.obitypes     cimport OBIIdx_NA
-
-from obitools3.utils cimport str2bytes
-
-from cpython.int cimport PyInt_FromSsize_t
-
-
-cdef class OBIDMS_column_idx(OBIDMS_column):
-
-    cpdef object get_item(self, size_t line_nb, str element_name):
-        cdef obiidx_t value
-        cdef object result
-        value = obi_column_get_obiidx_with_elt_name(self.pointer, line_nb, str2bytes(element_name))
-        if obi_errno > 0 :
-            raise IndexError(line_nb, element_name)
-        if value == OBIIdx_NA :
-            result = None
-        else :
-            result = PyInt_FromSsize_t(value)
-        return result
-    
-    cpdef set_item(self, size_t line_nb, str element_name, obiidx_t value):
-        raise Exception("Column is read-only")
-
-    cpdef close(self):
-        if obi_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
-                  
-
-cdef class OBIDMS_column_idx_writable(OBIDMS_column_idx):     
-
-    cpdef set_item(self, size_t line_nb, str element_name, obiidx_t value):
-        if obi_column_set_obiidx_with_elt_name(self.pointer, line_nb, str2bytes(element_name), value) < 0:
-            raise Exception("Problem setting a value in a column")
-
-    cpdef close(self):
-        if obi_truncate_and_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
- 

python/obitools3/obidms/_obidmscolumn_int.cfiles

@@ -1,16 +1,59 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
 ../../../src/obidmscolumn_int.c
 ../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
 ../../../src/obidmscolumndir.c
-../../../src/obidms.h
-../../../src/obidms.c
 ../../../src/obierrno.h
 ../../../src/obierrno.c
 ../../../src/obilittlebigman.h
 ../../../src/obilittlebigman.c
 ../../../src/obitypes.h
 ../../../src/obitypes.c
-../../../src/private_at_functions.h
-../../../src/private_at_functions.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
+../../../src/utils.h
+../../../src/utils.c

python/obitools3/obidms/_obidmscolumn_int.pxd

@@ -1,16 +1,14 @@
 #cython: language_level=3
 
-from .capi.obitypes cimport obiint_t
-from .capi.obidmscolumn cimport OBIDMS_column_p
-                                                
-from ._obidms cimport OBIDMS_column
+from .capi.obitypes cimport index_t
+from ._obidms       cimport OBIDMS_column, OBIDMS_column_multi_elts
 
 
 cdef class OBIDMS_column_int(OBIDMS_column):
-    cpdef object get_item(self, size_t line_nb, str element_name)
-    cpdef set_item(self, size_t line_nb, str element_name, obiint_t value)
-    cpdef close(self)
+    cpdef object get_line(self, index_t line_nb)
+    cpdef set_line(self, index_t line_nb, object value)
 
-cdef class OBIDMS_column_int_writable(OBIDMS_column_int):
-    cpdef set_item(self, size_t line_nb, str element_name, obiint_t value)
-    cpdef close(self)
+cdef class OBIDMS_column_multi_elts_int(OBIDMS_column_multi_elts):
+    cpdef object get_item(self, index_t line_nb, str element_name)
+    cpdef object get_line(self, index_t line_nb)
+    cpdef set_item(self, index_t line_nb, str element_name, object value)

python/obitools3/obidms/_obidmscolumn_int.pyx

@@ -1,25 +1,44 @@
 #cython: language_level=3
 
-from .capi.obidmscolumn cimport obi_close_column,\
-                                obi_truncate_and_close_column, \
-                                obi_column_get_obiint_with_elt_name, \
-                                obi_column_set_obiint_with_elt_name
+from .capi.obiview      cimport obi_column_get_obiint_with_elt_name_in_view, \
+                                obi_column_get_obiint_with_elt_idx_in_view, \
+                                obi_column_set_obiint_with_elt_name_in_view, \
+                                obi_column_set_obiint_with_elt_idx_in_view
 from .capi.obierrno     cimport obi_errno
-from .capi.obitypes     cimport OBIInt_NA
+from .capi.obitypes     cimport OBIInt_NA, obiint_t
 
 from obitools3.utils cimport str2bytes
 
 from cpython.int cimport PyInt_FromLong
 
-from ._obidms cimport OBIDMS_column
-
 
 cdef class OBIDMS_column_int(OBIDMS_column):
 
-    cpdef object get_item(self, size_t line_nb, str element_name):
+    cpdef object get_line(self, index_t line_nb):
         cdef obiint_t value
         cdef object result
-        value = obi_column_get_obiint_with_elt_name(self.pointer, line_nb, str2bytes(element_name))
+        value = obi_column_get_obiint_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0)
+        if obi_errno > 0 :
+            raise IndexError(line_nb)
+        if value == OBIInt_NA :
+            result = None
+        else :
+            result = PyInt_FromLong(value)
+        return result
+            
+    cpdef set_line(self, index_t line_nb, object value):
+        if value is None :
+            value = OBIInt_NA
+        if obi_column_set_obiint_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, <obiint_t> value) < 0:
+            raise Exception("Problem setting a value in a column")
+
+
+cdef class OBIDMS_column_multi_elts_int(OBIDMS_column_multi_elts):
+
+    cpdef object get_item(self, index_t line_nb, str element_name):
+        cdef obiint_t value
+        cdef object result
+        value = obi_column_get_obiint_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name))
         if obi_errno > 0 :
             raise IndexError(line_nb, element_name)
         if value == OBIInt_NA :
@@ -28,21 +47,32 @@ cdef class OBIDMS_column_int(OBIDMS_column):
             result = PyInt_FromLong(value)
         return result
     
-    cpdef set_item(self, size_t line_nb, str element_name, obiint_t value):
-        raise Exception("Column is read-only")
+    cpdef object get_line(self, index_t line_nb) :
+        cdef obiint_t value
+        cdef object value_in_result
+        cdef dict result
+        cdef index_t i
+        cdef bint all_NA
+        result = {}
+        all_NA = True
+        for i in range(self.nb_elements_per_line) :
+            value = obi_column_get_obiint_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, i)
+            if obi_errno > 0 :
+                raise IndexError(line_nb)
+            if value == OBIInt_NA :
+                value_in_result = None
+            else :
+                value_in_result = PyInt_FromLong(value)
+            result[self.elements_names[i]] = value_in_result
+            if all_NA and (value_in_result is not None) :
+                all_NA = False
+        if all_NA :
+            result = None   # TODO discuss
+        return result
 
-    cpdef close(self):
-        if obi_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
-                  
-
-cdef class OBIDMS_column_int_writable(OBIDMS_column_int):     
-
-    cpdef set_item(self, size_t line_nb, str element_name, obiint_t value):
-        if obi_column_set_obiint_with_elt_name(self.pointer, line_nb, str2bytes(element_name), value) < 0:
+    cpdef set_item(self, index_t line_nb, str element_name, object value):
+        if value is None :
+            value = OBIInt_NA
+        if obi_column_set_obiint_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), <obiint_t> value) < 0:
             raise Exception("Problem setting a value in a column")
 
-    cpdef close(self):
-        if obi_truncate_and_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
- 

python/obitools3/obidms/_obidmscolumn_qual.cfiles

@@ -0,0 +1,59 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
+../../../src/obidmscolumn.h
+../../../src/obidmscolumn.c
+../../../src/obidmscolumndir.h
+../../../src/obidmscolumndir.c
+../../../src/obierrno.h
+../../../src/obierrno.c
+../../../src/obilittlebigman.h
+../../../src/obilittlebigman.c
+../../../src/obitypes.h
+../../../src/obitypes.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
+../../../src/utils.h
+../../../src/utils.c

python/obitools3/obidms/_obidmscolumn_qual.pxd

@@ -0,0 +1,20 @@
+#cython: language_level=3
+
+from .capi.obitypes cimport index_t
+from ._obidms       cimport OBIDMS_column , OBIDMS_column_multi_elts
+
+
+cdef class OBIDMS_column_qual(OBIDMS_column):
+    cpdef object get_line(self, index_t line_nb)
+    cpdef object get_str_line(self, index_t line_nb)
+    cpdef set_line(self, index_t line_nb, object value)
+    cpdef set_str_line(self, index_t line_nb, object value)
+
+
+cdef class OBIDMS_column_multi_elts_qual(OBIDMS_column_multi_elts):
+    cpdef object get_item(self, index_t line_nb, str element_name)
+    cpdef object get_str_item(self, index_t line_nb, str element_name)
+    cpdef object get_line(self, index_t line_nb)
+    cpdef object get_str_line(self, index_t line_nb)
+    cpdef set_item(self, index_t line_nb, str element_name, object value)
+    cpdef set_str_item(self, index_t line_nb, str element_name, object value)

python/obitools3/obidms/_obidmscolumn_qual.pyx

@@ -0,0 +1,184 @@
+#cython: language_level=3
+ 
+from .capi.obiview      cimport obi_column_get_obiqual_char_with_elt_name_in_view, \
+                                obi_column_get_obiqual_char_with_elt_idx_in_view, \
+                                obi_column_set_obiqual_char_with_elt_name_in_view, \
+                                obi_column_set_obiqual_char_with_elt_idx_in_view, \
+                                obi_column_get_obiqual_int_with_elt_name_in_view, \
+                                obi_column_get_obiqual_int_with_elt_idx_in_view, \
+                                obi_column_set_obiqual_int_with_elt_name_in_view, \
+                                obi_column_set_obiqual_int_with_elt_idx_in_view
+                                
+from .capi.obierrno     cimport obi_errno
+from .capi.obitypes     cimport OBIQual_char_NA, OBIQual_int_NA, const_char_p
+
+from ._obidms           cimport OBIView
+
+from obitools3.utils cimport str2bytes, bytes2str
+
+from libc.stdlib cimport free
+from libc.string cimport strcmp
+from libc.stdint cimport uint8_t
+from libc.stdlib cimport malloc
+
+
+cdef class OBIDMS_column_qual(OBIDMS_column):
+
+    cpdef object get_line(self, index_t line_nb):
+        cdef const uint8_t*  value
+        cdef int             value_length
+        cdef object          result
+        cdef int             i
+        value = obi_column_get_obiqual_int_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, &value_length)
+        if obi_errno > 0 :
+            raise IndexError(line_nb)
+        if value == OBIQual_int_NA :
+            result = None
+        else :
+            result = []
+            for i in range(value_length) :
+                result.append(<int>value[i])            
+        return result
+    
+    cpdef object get_str_line(self, index_t line_nb):
+        cdef char*  value
+        cdef object result
+        cdef int    i
+        value = obi_column_get_obiqual_char_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0)
+        if obi_errno > 0 :
+            raise IndexError(line_nb)
+        if value == OBIQual_char_NA :
+            result = None
+        else :
+            result = bytes2str(value)
+            free(value)
+        return result
+
+    cpdef set_line(self, index_t line_nb, object value):
+        cdef uint8_t* value_b
+        cdef int      value_length
+        if value is None :
+            if obi_column_set_obiqual_int_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, OBIQual_int_NA, 0) < 0:
+                raise Exception("Problem setting a value in a column")
+        else :
+            value_length = len(value)
+            value_b = <uint8_t*> malloc(value_length * sizeof(uint8_t))
+            for i in range(value_length) :
+                value_b[i] = <uint8_t>value[i]
+            if obi_column_set_obiqual_int_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, value_b, value_length) < 0:
+                raise Exception("Problem setting a value in a column")
+            free(value_b)
+
+    cpdef set_str_line(self, index_t line_nb, object value):
+        if value is None :
+            if obi_column_set_obiqual_char_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, OBIQual_char_NA) < 0:
+                raise Exception("Problem setting a value in a column")
+        else :
+            if obi_column_set_obiqual_char_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, str2bytes(value)) < 0:
+                raise Exception("Problem setting a value in a column")
+
+
+cdef class OBIDMS_column_multi_elts_qual(OBIDMS_column_multi_elts):
+ 
+    cpdef object get_item(self, index_t line_nb, str element_name):
+        cdef const uint8_t*  value
+        cdef int             value_length
+        cdef object          result
+        cdef int             i
+        value = obi_column_get_obiqual_int_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), &value_length)
+        if obi_errno > 0 :
+            raise IndexError(line_nb, element_name)
+        if value == OBIQual_int_NA :
+            result = None
+        else :
+            result = []
+            for i in range(value_length) :
+                result.append(<int>value[i])            
+        return result
+ 
+    cpdef object get_str_item(self, index_t line_nb, str element_name):
+        cdef char*  value
+        cdef object result
+        value = obi_column_get_obiqual_char_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name))
+        if obi_errno > 0 :
+            raise IndexError(line_nb, element_name)
+        if value == OBIQual_char_NA :
+            result = None
+        else :
+            result = bytes2str(value)
+            free(value)
+        return result
+
+    cpdef object get_line(self, index_t line_nb) :
+        cdef const uint8_t*  value
+        cdef int             value_length
+        cdef object          value_in_result
+        cdef dict            result
+        cdef index_t         i
+        cdef int             j
+        cdef bint            all_NA
+        result = {}
+        all_NA = True
+        for i in range(self.nb_elements_per_line) :
+            value = obi_column_get_obiqual_int_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, i, &value_length)
+            if obi_errno > 0 :
+                raise IndexError(line_nb)
+            if value == OBIQual_int_NA :
+                value_in_result = None
+            else :
+                value_in_result = []
+                for j in range(value_length) :
+                    value_in_result.append(<int>value[j])            
+            result[self.elements_names[i]] = value_in_result
+            if all_NA and (value_in_result is not None) :
+                all_NA = False
+        if all_NA :
+            result = None
+        return result
+ 
+    cpdef object get_str_line(self, index_t line_nb) :
+        cdef char*   value
+        cdef object  value_in_result
+        cdef dict    result
+        cdef index_t i
+        cdef bint    all_NA
+        result = {}
+        all_NA = True
+        for i in range(self.nb_elements_per_line) :
+            value = obi_column_get_obiqual_char_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, i)
+            if obi_errno > 0 :
+                raise IndexError(line_nb)
+            if value == OBIQual_char_NA :
+                value_in_result = None
+            else :
+                value_in_result = bytes2str(value)
+                free(value)
+            result[self.elements_names[i]] = value_in_result
+            if all_NA and (value_in_result is not None) :
+                all_NA = False
+        if all_NA :
+            result = None
+        return result
+
+    cpdef set_item(self, index_t line_nb, str element_name, object value):
+        cdef uint8_t* value_b
+        cdef int      value_length
+        if value is None :
+            if obi_column_set_obiqual_int_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), OBIQual_int_NA, 0) < 0:
+                raise Exception("Problem setting a value in a column")
+        else :
+            value_length = len(value)
+            value_b = <uint8_t*> malloc(value_length * sizeof(uint8_t))
+            for i in range(value_length) :
+                value_b[i] = <uint8_t>value[i]
+            if obi_column_set_obiqual_int_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), value_b, value_length) < 0:
+                raise Exception("Problem setting a value in a column")
+            free(value_b)
+
+    cpdef set_str_item(self, index_t line_nb, str element_name, object value):
+        if value is None :
+            if obi_column_set_obiqual_char_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), OBIQual_char_NA) < 0:
+                raise Exception("Problem setting a value in a column")
+        else :
+            if obi_column_set_obiqual_char_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), str2bytes(value)) < 0:
+                raise Exception("Problem setting a value in a column")

python/obitools3/obidms/_obidmscolumn_seq.cfiles

@@ -0,0 +1,59 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
+../../../src/obidmscolumn.h
+../../../src/obidmscolumn.c
+../../../src/obidmscolumndir.h
+../../../src/obidmscolumndir.c
+../../../src/obierrno.h
+../../../src/obierrno.c
+../../../src/obilittlebigman.h
+../../../src/obilittlebigman.c
+../../../src/obitypes.h
+../../../src/obitypes.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
+../../../src/utils.h
+../../../src/utils.c

python/obitools3/obidms/_obidmscolumn_seq.pxd

@@ -0,0 +1,26 @@
+#cython: language_level=3
+
+from .capi.obitypes cimport index_t
+from ._obidms       cimport OBIView, OBIDMS_column, OBIDMS_column_multi_elts
+
+
+cdef class OBIDMS_column_seq(OBIDMS_column):
+    cpdef object get_line(self, index_t line_nb)
+    cpdef set_line(self, index_t line_nb, object value)
+    
+    # TO DISCUSS : 
+    # I'am not sure that this method has to be declared here
+    # Alignment must be declared outside of the sequence object
+    cpdef align(self, 
+                OBIView score_view,
+                OBIDMS_column score_column, 
+                double threshold = *, 
+                bint normalize = *, 
+                int reference = *,
+                bint similarity_mode = *)
+
+
+cdef class OBIDMS_column_multi_elts_seq(OBIDMS_column_multi_elts):
+    cpdef object get_item(self, index_t line_nb, str element_name)
+    cpdef object get_line(self, index_t line_nb)
+    cpdef set_item(self, index_t line_nb, str element_name, object value)

python/obitools3/obidms/_obidmscolumn_seq.pyx

@@ -0,0 +1,127 @@
+#cython: language_level=3
+ 
+from .capi.obiview      cimport obi_column_get_obiseq_with_elt_name_in_view, \
+                                obi_column_get_obiseq_with_elt_idx_in_view, \
+                                obi_column_set_obiseq_with_elt_name_in_view, \
+                                obi_column_set_obiseq_with_elt_idx_in_view
+from .capi.obialign     cimport obi_align_one_column
+from .capi.obierrno     cimport obi_errno
+from .capi.obitypes     cimport OBISeq_NA, const_char_p
+
+from ._obidms           cimport OBIView
+
+from obitools3.utils cimport str2bytes, bytes2str
+
+from libc.stdlib cimport free
+
+
+cdef class OBIDMS_column_seq(OBIDMS_column):
+
+    cpdef object get_line(self, index_t line_nb):
+        cdef char*  value
+        cdef object result
+        value = obi_column_get_obiseq_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0)
+        if obi_errno > 0 :
+            raise IndexError(line_nb)
+        if value == OBISeq_NA :
+            result = None
+        else :
+            try:
+                result = <bytes> value
+            finally:
+                free(value)
+        return result
+
+    cpdef set_line(self, index_t line_nb, object value):
+        cdef bytes value_b
+
+        if value is None :
+            value_b = OBISeq_NA
+        elif isinstance(value, bytes) :
+            value_b = value
+        elif isinstance(value, str) :
+            value_b = str2bytes(value)
+        else:
+            raise TypeError('Sequence value must be of type Bytes, Str or None')
+
+        if obi_column_set_obiseq_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, value_b) < 0:
+            raise Exception("Problem setting a value in a column")
+        else :
+            if obi_column_set_obiseq_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, str2bytes(value)) < 0:
+                raise Exception("Problem setting a value in a column")
+    
+    # TODO choose alignment type (lcs or other) with supplementary argument
+    cpdef align(self, 
+                OBIView score_view,
+                OBIDMS_column score_column, 
+                double threshold = 0.0, 
+                bint normalize = True, 
+                int reference = 0,      # TODO
+                bint similarity_mode = True):
+        if (obi_align_one_column(self.view.pointer, (self.pointer)[0], score_view.pointer, (score_column.pointer)[0], threshold, normalize, reference, similarity_mode) < 0) :
+            raise Exception("An error occurred while aligning sequences")
+
+        
+
+cdef class OBIDMS_column_multi_elts_seq(OBIDMS_column_multi_elts):
+
+    cpdef object get_item(self, index_t line_nb, str element_name):
+        cdef char*  value
+        cdef object result
+        value = obi_column_get_obiseq_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name))
+        if obi_errno > 0 :
+            raise IndexError(line_nb, element_name)
+        if value == OBISeq_NA :
+            result = None
+        else :
+            try:
+                result = <bytes> value
+            finally:
+                free(value)
+        return result
+
+    
+    cpdef object get_line(self, index_t line_nb) :
+        cdef char*   value
+        cdef object  value_in_result
+        cdef dict    result
+        cdef index_t i
+        cdef bint    all_NA
+        result = {}
+        all_NA = True
+        for i in range(self.nb_elements_per_line) :
+            value = obi_column_get_obiseq_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, i)
+            if obi_errno > 0 :
+                raise IndexError(line_nb)
+            if value == OBISeq_NA :
+                value_in_result = None
+            else :
+                try:
+                    value_in_result = <bytes> value
+                finally:
+                    free(value)    
+            result[self.elements_names[i]] = value_in_result
+            if all_NA and (value_in_result is not None) :
+                all_NA = False
+        if all_NA :
+            result = None
+        return result
+
+    cpdef set_item(self, index_t line_nb, str element_name, object value):
+        cdef bytes value_b
+        
+        if value is None :
+            value_b = OBISeq_NA
+        elif isinstance(value, bytes) :
+            value_b = value
+        elif isinstance(value, str) :
+            value_b = str2bytes(value)
+        else:
+            raise TypeError('Sequence value must be of type Bytes, Str or None')
+        
+        if obi_column_set_obiseq_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), value_b) < 0:
+            raise Exception("Problem setting a value in a column")
+    
+#    cpdef align(self, ):        # TODO
+#        raise Exception("Columns with multiple sequences per line can't be aligned")    # TODO discuss
+

python/obitools3/obidms/_obidmscolumn_str.cfiles

@@ -0,0 +1,59 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
+../../../src/obidmscolumn.h
+../../../src/obidmscolumn.c
+../../../src/obidmscolumndir.h
+../../../src/obidmscolumndir.c
+../../../src/obierrno.h
+../../../src/obierrno.c
+../../../src/obilittlebigman.h
+../../../src/obilittlebigman.c
+../../../src/obitypes.h
+../../../src/obitypes.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
+../../../src/utils.h
+../../../src/utils.c

python/obitools3/obidms/_obidmscolumn_str.pxd

@@ -0,0 +1,14 @@
+#cython: language_level=3
+
+from .capi.obitypes cimport index_t
+from ._obidms       cimport OBIDMS_column, OBIDMS_column_multi_elts
+
+
+cdef class OBIDMS_column_str(OBIDMS_column):
+    cpdef object get_line(self, index_t line_nb)
+    cpdef set_line(self, index_t line_nb, object value)
+
+cdef class OBIDMS_column_multi_elts_str(OBIDMS_column_multi_elts):
+    cpdef object get_item(self, index_t line_nb, str element_name)
+    cpdef object get_line(self, index_t line_nb)
+    cpdef set_item(self, index_t line_nb, str element_name, object value)

python/obitools3/obidms/_obidmscolumn_str.pyx

@@ -0,0 +1,84 @@
+#cython: language_level=3
+ 
+from .capi.obiview      cimport obi_column_get_obistr_with_elt_name_in_view, \
+                                obi_column_get_obistr_with_elt_idx_in_view, \
+                                obi_column_set_obistr_with_elt_name_in_view, \
+                                obi_column_set_obistr_with_elt_idx_in_view
+from .capi.obierrno     cimport obi_errno
+from .capi.obitypes     cimport OBIStr_NA, const_char_p
+
+from obitools3.utils cimport str2bytes, bytes2str
+
+
+cdef class OBIDMS_column_str(OBIDMS_column):
+
+    cpdef object get_line(self, index_t line_nb):
+        cdef const_char_p value
+        cdef object       result
+        value = obi_column_get_obistr_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0)
+        if obi_errno > 0 :
+            raise IndexError(line_nb)
+        if value == OBIStr_NA :
+            result = None
+        else :
+            result = bytes2str(value)
+        # NOTE: value is not freed because the pointer points to a mmapped region in an AVL data file. (TODO discuss)
+        return result
+
+    cpdef set_line(self, index_t line_nb, object value):
+        if value is None :
+            if obi_column_set_obistr_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, OBIStr_NA) < 0:
+                raise Exception("Problem setting a value in a column")
+        else :
+            if obi_column_set_obistr_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, str2bytes(value)) < 0:
+                raise Exception("Problem setting a value in a column")
+
+
+cdef class OBIDMS_column_multi_elts_str(OBIDMS_column_multi_elts):
+
+    cpdef object get_item(self, index_t line_nb, str element_name):
+        cdef const_char_p value
+        cdef object       result
+        value = obi_column_get_obistr_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name))
+        if obi_errno > 0 :
+            raise IndexError(line_nb, element_name)
+        if value == OBIStr_NA :
+            result = None
+        else :
+            result = bytes2str(value)
+        # NOTE: value is not freed because the pointer points to a mmapped region in an AVL data file. (TODO discuss)
+        return result
+
+    cpdef object get_line(self, index_t line_nb) :
+        cdef const_char_p value
+        cdef object       value_in_result
+        cdef dict         result
+        cdef index_t      i
+        cdef bint         all_NA
+        result = {}
+        all_NA = True
+        for i in range(self.nb_elements_per_line) :
+            value = obi_column_get_obistr_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, i)
+            if obi_errno > 0 :
+                raise IndexError(line_nb)
+            if value == OBIStr_NA :
+                value_in_result = None
+            else :
+                value_in_result = bytes2str(value)
+            # NOTE: value is not freed because the pointer points to a mmapped region in an AVL data file. (TODO discuss)
+            result[self.elements_names[i]] = value_in_result
+            if all_NA and (value_in_result is not None) :
+                all_NA = False
+        if all_NA :
+            result = None
+        return result
+
+    cpdef set_item(self, index_t line_nb, str element_name, object value):
+        cdef bytes value_b
+        if value is None :
+            value_b = OBIStr_NA
+        else :
+            value_b = str2bytes(value)
+        if obi_column_set_obistr_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), value_b) < 0:
+            raise Exception("Problem setting a value in a column")
+    

python/obitools3/obidms/_obiseq.cfiles

@@ -0,0 +1,59 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
+../../../src/obidmscolumn.h
+../../../src/obidmscolumn.c
+../../../src/obidmscolumndir.h
+../../../src/obidmscolumndir.c
+../../../src/obierrno.h
+../../../src/obierrno.c
+../../../src/obilittlebigman.h
+../../../src/obilittlebigman.c
+../../../src/obitypes.h
+../../../src/obitypes.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
+../../../src/utils.h
+../../../src/utils.c

python/obitools3/obidms/_obiseq.pxd

@@ -0,0 +1,38 @@
+#cython: language_level=3
+
+from ._obidms cimport OBIView_line
+
+
+cdef class OBI_Seq(dict) :
+    cdef object id
+    cdef object definition
+    cdef object sequence
+
+    cpdef set_id(self, object id)
+    cpdef object get_id(self)
+    cpdef set_definition(self, object definition)
+    cpdef object get_definition(self)
+    cpdef object get_sequence(self)
+
+
+cdef class OBI_Nuc_Seq(OBI_Seq) :
+    cdef object quality
+    
+    #cpdef object reverse_complement(self)    
+    cpdef set_sequence(self, object sequence)
+    cpdef set_quality(self, object quality)
+    cpdef object get_quality(self)
+
+
+cdef class OBI_Nuc_Seq_Stored(OBIView_line) :
+    cpdef set_id(self, object id)
+    cpdef object get_id(self)
+    cpdef set_definition(self, object definition)
+    cpdef object get_definition(self)
+    cpdef set_sequence(self, object sequence)
+    cpdef object get_sequence(self)
+    cpdef set_quality(self, object quality)
+    cpdef object get_quality(self)
+    cpdef object get_str_quality(self)
+
+   # cpdef object reverse_complement(self)

python/obitools3/obidms/_obiseq.pyx

@@ -0,0 +1,97 @@
+#cython: language_level=3
+
+from obitools3.utils cimport bytes2str, str2bytes
+
+from .capi.obiview cimport NUC_SEQUENCE_COLUMN, \
+                           ID_COLUMN, \
+                           DEFINITION_COLUMN, \
+                           QUALITY_COLUMN
+
+cdef class OBI_Seq(dict) :
+
+    def __init__(self, object id, object seq, object definition=None) :
+        self.set_id(id)
+        self.set_sequence(seq)
+        if definition is not None :
+            self.set_definition(definition)
+        
+    cpdef set_id(self, object id) :
+        self.id = id
+        self[bytes2str(ID_COLUMN)] = id
+        
+    cpdef get_id(self) :
+        return self.id
+            
+    cpdef set_definition(self, object definition) :
+        self.definition = definition
+        self[bytes2str(DEFINITION_COLUMN)] = definition
+        
+    cpdef get_definition(self) :
+        return self.definition
+
+    cpdef get_sequence(self) :
+        return self.sequence
+
+    def __str__(self) :
+        return self.sequence    # or not
+
+
+cdef class OBI_Nuc_Seq(OBI_Seq) :
+    
+    cpdef set_sequence(self, object sequence) :
+        self.sequence = sequence
+        self[bytes2str(NUC_SEQUENCE_COLUMN)] = sequence
+
+    cpdef set_quality(self, object quality) :
+        self.quality = quality
+        self[bytes2str(QUALITY_COLUMN)] = quality
+
+    cpdef get_quality(self) :
+        return self.quality
+
+#    cpdef str reverse_complement(self) :    TODO in C ?
+#        pass
+
+
+cdef class OBI_Nuc_Seq_Stored(OBIView_line) :
+
+    # TODO store the str version of column name macros
+
+    cpdef set_id(self, object id) :
+        self[bytes2str(ID_COLUMN)] = id
+        
+    cpdef object get_id(self) :
+        return self[bytes2str(ID_COLUMN)]
+            
+    cpdef set_definition(self, object definition) :
+        self[bytes2str(DEFINITION_COLUMN)] = definition
+
+    cpdef object get_definition(self) :
+        return self[bytes2str(DEFINITION_COLUMN)]
+
+    cpdef set_sequence(self, object sequence) :
+        self[bytes2str(NUC_SEQUENCE_COLUMN)] = sequence
+
+    cpdef object get_sequence(self) :
+        return self[bytes2str(NUC_SEQUENCE_COLUMN)]
+    
+    cpdef set_quality(self, object quality) :
+        if (type(quality) == list) or (quality is None) :
+            self[bytes2str(QUALITY_COLUMN)] = quality
+        else :  # Quality is in str form
+            (((self.view).columns)[bytes2str(QUALITY_COLUMN)]).set_str_line(self.index, quality)
+
+    cpdef object get_quality(self) :
+        return self[bytes2str(QUALITY_COLUMN)]
+
+    cpdef object get_str_quality(self) :
+        return ((self.view).columns)[bytes2str(QUALITY_COLUMN)].get_str_line(self.index)
+
+#    def __str__(self) :
+#        return self[bytes2str(NUC_SEQUENCE_COLUMN)]    # or not
+    
+#    cpdef str reverse_complement(self) :    TODO in C ?
+#        pass
+
+    # TODO static method to import?
+    

python/obitools3/obidms/_obitaxo.cfiles

@@ -0,0 +1,59 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
+../../../src/obidmscolumn.h
+../../../src/obidmscolumn.c
+../../../src/obidmscolumndir.h
+../../../src/obidmscolumndir.c
+../../../src/obierrno.h
+../../../src/obierrno.c
+../../../src/obilittlebigman.h
+../../../src/obilittlebigman.c
+../../../src/obitypes.h
+../../../src/obitypes.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
+../../../src/utils.h
+../../../src/utils.c

python/obitools3/obidms/_obitaxo.pxd

@@ -0,0 +1,31 @@
+#cython: language_level=3
+
+from .capi.obitaxonomy cimport ecotx_t, OBIDMS_taxonomy_p
+
+from libc.stdint   cimport int32_t
+
+
+cdef class OBI_Taxonomy :
+
+    cdef str name
+    cdef OBIDMS_taxonomy_p pointer
+    
+    cpdef close(self)
+
+ 
+cdef class OBI_Taxon :
+
+    cdef ecotx_t* pointer
+    cdef int32_t  taxid
+    cdef int32_t  rank
+    cdef int32_t  farest
+    cdef ecotx_t* parent
+    cdef str name
+        
+    cpdef int32_t taxid(self)
+    
+    cpdef int32_t rank(self)
+    
+    cpdef int32_t farest(self)
+    
+    cpdef OBI_Taxon parent(self)

python/obitools3/obidms/_obitaxo.pyx

@@ -0,0 +1,65 @@
+#cython: language_level=3
+
+from obitools3.utils   cimport bytes2str, str2bytes
+
+from .capi.obitaxonomy cimport obi_read_taxonomy, \
+                               obi_close_taxonomy, \
+                               obi_taxo_get_taxon_with_taxid
+
+from ._obidms cimport OBIDMS
+
+from cpython.pycapsule cimport PyCapsule_New, PyCapsule_GetPointer
+
+
+cdef class OBI_Taxonomy :
+
+    def __init__(self, OBIDMS dms, str name) :
+        
+        self.name = name
+        self.pointer = obi_read_taxonomy(dms.pointer, str2bytes(name), True)    # TODO discuss
+    
+    
+    def __getitem__(self, object ref):
+        
+        cdef ecotx_t* taxon_p
+        cdef object taxon_capsule
+        
+        if type(ref) == int :
+            taxon_p = obi_taxo_get_taxon_with_taxid(self.pointer, ref)
+            taxon_capsule = PyCapsule_New(taxon_p, NULL, NULL)
+            return OBI_Taxon(taxon_capsule)
+    
+         
+    cpdef close(self) :
+        if (obi_close_taxonomy(self.pointer) < 0) :
+            raise Exception("Error closing the taxonomy")
+        
+
+cdef class OBI_Taxon :    # dict subclass?
+
+    def __init__(self, object taxon_capsule) :
+        
+        cdef ecotx_t* taxon
+        
+        taxon = <ecotx_t*> PyCapsule_GetPointer(taxon_capsule, NULL)   
+        self.pointer = taxon
+        self.taxid = taxon.taxid
+        self.rank = taxon.rank
+        self.farest = taxon.farest
+        self.parent = taxon.parent
+        self.name = bytes2str(taxon.name)
+        
+    cpdef int32_t taxid(self):
+        return self.taxid
+    
+    cpdef int32_t rank(self):
+        return self.rank
+    
+    cpdef int32_t farest(self):
+        return self.farest
+    
+    cpdef OBI_Taxon parent(self):
+        cdef object parent_capsule
+        parent_capsule = PyCapsule_New(self.parent, NULL, NULL)
+        return OBI_Taxon(parent_capsule)
+    

python/obitools3/obidms/capi/obialign.pxd

@@ -0,0 +1,10 @@
+#cython: language_level=3
+
+from ..capi.obiview      cimport Obiview_p
+from ..capi.obidmscolumn cimport OBIDMS_column_p
+
+
+cdef extern from "obi_align.h" nogil:
+
+    int obi_align_one_column(Obiview_p seq_view, OBIDMS_column_p seq_column, Obiview_p score_view, OBIDMS_column_p score_column, double threshold, bint normalize, int reference, bint similarity_mode)
+

python/obitools3/obidms/capi/obidmscolumn.pxd

@@ -8,106 +8,234 @@ from ..capi.obitypes cimport const_char_p, \
                              obibool_t, \
                              obichar_t, \
                              obifloat_t, \
-                             obiidx_t
+                             index_t, \
+                             time_t
+
+from libc.stdint cimport uint8_t
+
 
 cdef extern from "obidmscolumn.h" nogil:
+    
+    struct OBIDMS_column_header_t:
+        size_t          header_size
+        size_t          data_size
+        index_t         line_count
+        index_t         lines_used
+        index_t         nb_elements_per_line
+        const_char_p    elements_names
+        OBIType_t       returned_data_type
+        OBIType_t       stored_data_type
+        time_t          creation_date
+        obiversion_t    version
+        obiversion_t    cloned_from
+        const_char_p    name
+        const_char_p    indexer_name
+        const_char_p    comments
+    
+    ctypedef OBIDMS_column_header_t* OBIDMS_column_header_p
+
     struct OBIDMS_column_t:
-        pass
+        OBIDMS_p               dms
+        OBIDMS_column_header_p header
+        bint                   writable
         
     ctypedef OBIDMS_column_t* OBIDMS_column_p
 
     OBIDMS_column_p obi_create_column(OBIDMS_p dms, 
                                       const_char_p column_name, 
                                       OBIType_t type, 
-                                      size_t nb_lines, 
-                                      size_t nb_elements_per_line, 
-                                      const_char_p elements_names)
-    
-    size_t obi_column_get_nb_lines_used(OBIDMS_column_p column)
-    
-    const_char_p obi_column_get_elements_names(OBIDMS_column_p column)
-    
-    void obi_column_make_unwritable(OBIDMS_column_p column)
+                                      index_t nb_lines, 
+                                      index_t nb_elements_per_line, 
+                                      const_char_p elements_names,
+                                      const_char_p indexer_name,
+                                      const_char_p comments)
                 
     OBIDMS_column_p obi_open_column(OBIDMS_p dms, 
                                     const_char_p column_name, 
                                     obiversion_t version_number)
         
-    OBIType_t obi_column_get_type(OBIDMS_column_p column)
-    
     int obi_close_column(OBIDMS_column_p column)
         
-    OBIType_t obi_column_get_data_type_from_name(OBIDMS_p dms, 
-                                                 const_char_p column_name)
-    
     OBIDMS_column_p obi_clone_column(OBIDMS_p dms, 
+                                     OBIDMS_column_p line_selection,
                                      const_char_p column_name, 
                                      obiversion_t version_number, 
                                      bint clone_data)
     
-    int obi_truncate_and_close_column(OBIDMS_column_p column)
+    int obi_close_column(OBIDMS_column_p column)
     
     obiversion_t obi_column_get_latest_version_from_name(OBIDMS_p dms, 
                                                          const_char_p column_name)
         
-    OBIType_t obi_column_get_data_type_from_name(OBIDMS_p dms, 
-                                                 const_char_p column_name)
+    OBIDMS_column_header_p obi_column_get_header_from_name(OBIDMS_p dms, 
+                                                           const_char_p column_name,
+                                                           obiversion_t version_number)
     
-    size_t obi_column_get_line_count_from_name(OBIDMS_p dms, 
-                                               const_char_p column_name)
+    int obi_close_header(OBIDMS_column_header_p header)
+
+    int obi_select(OBIDMS_column_p line_selection_column, index_t line_to_grep)
 
 
 cdef extern from "obidmscolumn_int.h" nogil:
+
     int obi_column_set_obiint_with_elt_name(OBIDMS_column_p column, 
-                                            size_t line_nb, 
+                                            index_t line_nb, 
                                             const_char_p element_name, 
                                             obiint_t value)
 
+    int obi_column_set_obiint_with_elt_idx(OBIDMS_column_p column, 
+                                           index_t line_nb, 
+                                           index_t element_idx, 
+                                           obiint_t value)
+    
     obiint_t obi_column_get_obiint_with_elt_name(OBIDMS_column_p column, 
-                                                 size_t line_nb, 
+                                                 index_t line_nb, 
                                                  const_char_p element_name)
     
+    obiint_t obi_column_get_obiint_with_elt_idx(OBIDMS_column_p column, 
+                                                index_t line_nb, 
+                                                index_t element_idx)
+
 cdef extern from "obidmscolumn_bool.h" nogil:
 
     int obi_column_set_obibool_with_elt_name(OBIDMS_column_p column, 
-                                             size_t line_nb, 
+                                             index_t line_nb, 
                                              const_char_p element_name, 
                                              obibool_t value)
 
+    int obi_column_set_obibool_with_elt_idx(OBIDMS_column_p column, 
+                                            index_t line_nb, 
+                                            index_t element_idx, 
+                                            obibool_t value)
+    
     obibool_t obi_column_get_obibool_with_elt_name(OBIDMS_column_p column, 
-                                                   size_t line_nb, 
+                                                   index_t line_nb, 
                                                    const_char_p element_name)
     
+    obibool_t obi_column_get_obibool_with_elt_idx(OBIDMS_column_p column, 
+                                                  index_t line_nb, 
+                                                  index_t element_idx)
+
 cdef extern from "obidmscolumn_char.h" nogil:
 
     int obi_column_set_obichar_with_elt_name(OBIDMS_column_p column, 
-                                             size_t line_nb, 
+                                             index_t line_nb, 
                                              const_char_p element_name, 
                                              obichar_t value)
 
+    int obi_column_set_obichar_with_elt_idx(OBIDMS_column_p column, 
+                                            index_t line_nb, 
+                                            index_t element_idx, 
+                                            obichar_t value)
+    
     obichar_t obi_column_get_obichar_with_elt_name(OBIDMS_column_p column, 
-                                                   size_t line_nb, 
+                                                   index_t line_nb, 
                                                    const_char_p element_name)
     
+    obichar_t obi_column_get_obichar_with_elt_idx(OBIDMS_column_p column, 
+                                                  index_t line_nb, 
+                                                  index_t element_idx)
+
 cdef extern from "obidmscolumn_float.h" nogil:
 
     int obi_column_set_obifloat_with_elt_name(OBIDMS_column_p column, 
-                                              size_t line_nb, 
+                                              index_t line_nb, 
                                               const_char_p element_name, 
                                               obifloat_t value)
 
+    int obi_column_set_obifloat_with_elt_idx(OBIDMS_column_p column, 
+                                             index_t line_nb, 
+                                             index_t element_idx, 
+                                             obifloat_t value)
+    
     obifloat_t obi_column_get_obifloat_with_elt_name(OBIDMS_column_p column, 
-                                                     size_t line_nb, 
+                                                     index_t line_nb, 
                                                      const_char_p element_name)
     
-cdef extern from "obidmscolumn_idx.h" nogil:
+    obifloat_t obi_column_get_obifloat_with_elt_idx(OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    index_t element_idx)
 
-    int obi_column_set_obiidx_with_elt_name(OBIDMS_column_p column, 
-                                            size_t line_nb, 
+cdef extern from "obidmscolumn_str.h" nogil:
+
+    int obi_column_set_obistr_with_elt_name(OBIDMS_column_p column, 
+                                            index_t line_nb, 
                                             const_char_p element_name, 
-                                            obiidx_t value)
+                                            const_char_p value)
 
-    obiidx_t obi_column_get_obiidx_with_elt_name(OBIDMS_column_p column, 
-                                                 size_t line_nb, 
-                                                 const_char_p element_name)
+    int obi_column_set_obistr_with_elt_idx(OBIDMS_column_p column, 
+                                           index_t line_nb, 
+                                           index_t element_idx, 
+                                           const_char_p value)
+    
+    const_char_p obi_column_get_obistr_with_elt_name(OBIDMS_column_p column, 
+                                                     index_t line_nb, 
+                                                     const_char_p element_name)
+    
+    const_char_p obi_column_get_obistr_with_elt_idx(OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    index_t element_idx)
+
+cdef extern from "obidmscolumn_seq.h" nogil:
+
+    int obi_column_set_obiseq_with_elt_name(OBIDMS_column_p column, 
+                                            index_t line_nb, 
+                                            const_char_p element_name, 
+                                            const_char_p value)
+
+    int obi_column_set_obiseq_with_elt_idx(OBIDMS_column_p column, 
+                                           index_t line_nb, 
+                                           index_t element_idx, 
+                                           const_char_p value)
+    
+    char* obi_column_get_obiseq_with_elt_name(OBIDMS_column_p column, 
+                                              index_t line_nb, 
+                                              const_char_p element_name)
+    
+    char* obi_column_get_obiseq_with_elt_idx(OBIDMS_column_p column, 
+                                             index_t line_nb, 
+                                             index_t element_idx)
+
+
+cdef extern from "obidmscolumn_qual.h" nogil:
+
+    int obi_column_set_obiqual_char_with_elt_name(OBIDMS_column_p column, 
+                                                  index_t line_nb, 
+                                                  const_char_p element_name, 
+                                                  const_char_p value)
+
+    int obi_column_set_obiqual_char_with_elt_idx(OBIDMS_column_p column, 
+                                                 index_t line_nb, 
+                                                 index_t element_idx, 
+                                                 const_char_p value)
+    
+    int obi_column_set_obiqual_int_with_elt_name(OBIDMS_column_p column, 
+                                                  index_t line_nb, 
+                                                  const_char_p element_name, 
+                                                  const uint8_t* value,
+                                                  int value_length)
+    
+    int obi_column_set_obiqual_int_with_elt_idx(OBIDMS_column_p column, 
+                                                index_t line_nb, 
+                                                index_t element_idx, 
+                                                const uint8_t* value,
+                                                int value_length)
+    
+    char* obi_column_get_obiqual_char_with_elt_name(OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    const_char_p element_name)
+    
+    char* obi_column_get_obiqual_char_with_elt_idx(OBIDMS_column_p column, 
+                                                   index_t line_nb, 
+                                                   index_t element_idx)
+
+    const uint8_t* obi_column_get_obiqual_int_with_elt_name(OBIDMS_column_p column, 
+                                                            index_t line_nb, 
+                                                            const_char_p element_name,
+                                                            int* value_length)
+    
+    const uint8_t* obi_column_get_obiqual_int_with_elt_idx(OBIDMS_column_p column, 
+                                                   index_t line_nb, 
+                                                   index_t element_idx,
+                                                   int* value_length)
 

python/obitools3/obidms/capi/obitaxonomy.pxd

@@ -0,0 +1,42 @@
+#cython: language_level=3
+
+from .obitypes     cimport const_char_p
+from .obidms       cimport OBIDMS_p
+from libc.stdint   cimport int32_t
+
+
+cdef extern from "obidms_taxonomy.h" nogil:
+
+    struct OBIDMS_taxonomy_t
+    ctypedef OBIDMS_taxonomy_t* OBIDMS_taxonomy_p
+    
+    struct ecotxnode :
+        int32_t           taxid
+        int32_t           rank
+        int32_t           farest
+        ecotxnode*        parent
+        char*             name
+    
+    ctypedef ecotxnode ecotx_t
+
+
+    OBIDMS_taxonomy_p obi_read_taxonomy(OBIDMS_p dms, const_char_p taxonomy_name, bint read_alternative_names)
+    
+    int obi_close_taxonomy(OBIDMS_taxonomy_p taxonomy)
+    
+    ecotx_t* obi_taxo_get_parent_at_rank(ecotx_t* taxon, int32_t rankidx)
+    
+    ecotx_t* obi_taxo_get_taxon_with_taxid(OBIDMS_taxonomy_p taxonomy, int32_t taxid)
+    
+    bint obi_taxo_is_taxon_under_taxid(ecotx_t* taxon, int32_t other_taxid)
+    
+    ecotx_t* obi_taxo_get_species(ecotx_t* taxon, OBIDMS_taxonomy_p taxonomy)
+    
+    ecotx_t* obi_taxo_get_genus(ecotx_t* taxon, OBIDMS_taxonomy_p taxonomy)
+    
+    ecotx_t* obi_taxo_get_family(ecotx_t* taxon, OBIDMS_taxonomy_p taxonomy)
+    
+    ecotx_t* obi_taxo_get_kingdom(ecotx_t* taxon, OBIDMS_taxonomy_p taxonomy)
+    
+    ecotx_t* obi_taxo_get_superkingdom(ecotx_t* taxon, OBIDMS_taxonomy_p taxonomy)
+    

python/obitools3/obidms/capi/obitypes.pxd

@@ -1,28 +1,33 @@
 #cython: language_level=3
 
 
-from libc.stdint cimport int32_t
+from libc.stdint cimport int32_t, int64_t, uint8_t
+
+from posix.types cimport time_t
 
 
 cdef extern from *:
     ctypedef char* const_char_p "const char*"
 
 
-cdef extern from "obidmscolumn.h" nogil:
-
-    ctypedef int32_t obiversion_t
+cdef extern from "encode.h" nogil:
+    bint only_ATGC(const_char_p seq)
 
 
 cdef extern from "obitypes.h" nogil:
 
-    enum OBIType:       # TODO je sais pas si ça sert de declarer le contenu de l'enum
-        OBI_VOID = 0,
+    enum OBIType:
+        OBI_VOID,
         OBI_INT,
         OBI_FLOAT,
         OBI_BOOL,
         OBI_CHAR,
+        OBI_QUAL,
+        OBI_STR,
+        OBI_SEQ,
         OBI_IDX
 
+    
     ctypedef OBIType OBIType_t
     
     enum OBIBool:
@@ -30,14 +35,21 @@ cdef extern from "obitypes.h" nogil:
     
     ctypedef OBIBool obibool_t
     ctypedef int32_t obiint_t
-    ctypedef double obifloat_t
-    ctypedef char obichar_t  
-    ctypedef size_t obiidx_t
+    ctypedef double  obifloat_t
+    ctypedef char    obichar_t  
+    ctypedef int64_t index_t
 
-    extern obiint_t   OBIInt_NA
-    extern obiidx_t   OBIIdx_NA
-    extern obifloat_t OBIFloat_NA
-    extern obichar_t  OBIChar_NA
-    extern obibool_t  OBIBool_NA
+    ctypedef int32_t obiversion_t
+
+    extern obiint_t      OBIInt_NA
+    extern index_t       OBIIdx_NA
+    extern obifloat_t    OBIFloat_NA
+    extern obichar_t     OBIChar_NA
+    extern obibool_t     OBIBool_NA
+    extern const_char_p  OBISeq_NA
+    extern const_char_p  OBIStr_NA
+    extern const_char_p  OBIQual_char_NA
+    extern uint8_t*      OBIQual_int_NA
+
+    const_char_p name_data_type(int data_type)
 
-    char* name_data_type(int data_type)

python/obitools3/obidms/capi/obiutils.pxd

@@ -0,0 +1,12 @@
+#cython: language_level=3
+
+
+from posix.types cimport time_t
+
+from ..capi.obitypes cimport const_char_p
+
+
+cdef extern from "utils.h" nogil:
+
+    const_char_p obi_format_date(time_t date)
+

python/obitools3/obidms/capi/obiview.pxd

@@ -0,0 +1,280 @@
+#cython: language_level=3
+
+from .obitypes           cimport const_char_p, \
+                                 OBIType_t, \
+                                 obiversion_t, \
+                                 obiint_t, \
+                                 obibool_t, \
+                                 obichar_t, \
+                                 obifloat_t, \
+                                 index_t, \
+                                 time_t
+from ..capi.obidms       cimport OBIDMS_p
+from ..capi.obidmscolumn cimport OBIDMS_column_p
+
+from libc.stdint cimport uint8_t
+
+
+cdef extern from "obiview.h" nogil:
+
+    extern const_char_p VIEW_TYPE_NUC_SEQS
+    extern const_char_p NUC_SEQUENCE_COLUMN
+    extern const_char_p ID_COLUMN
+    extern const_char_p DEFINITION_COLUMN
+    extern const_char_p QUALITY_COLUMN
+
+
+    struct Column_reference_t :
+        const_char_p  column_name
+        obiversion_t  version
+    
+    ctypedef Column_reference_t* Column_reference_p
+
+
+    struct Obiview_infos_t :
+        time_t              creation_date
+        const_char_p        name
+        const_char_p        created_from
+        const_char_p        view_type
+        bint                all_lines
+        Column_reference_t  line_selection
+        index_t             line_count
+        int                 column_count
+        Column_reference_p  column_references
+        const_char_p        comments
+
+    ctypedef Obiview_infos_t* Obiview_infos_p
+
+
+    struct Obiview_t :
+        Obiview_infos_p     infos
+        OBIDMS_p            dms
+        bint                read_only
+        OBIDMS_column_p     line_selection
+        OBIDMS_column_p     new_line_selection
+        OBIDMS_column_p     columns
+
+    ctypedef Obiview_t* Obiview_p
+
+
+    Obiview_p obi_new_view_nuc_seqs(OBIDMS_p dms, const_char_p view_name, Obiview_p view_to_clone, index_t* line_selection, const_char_p comments)
+
+    Obiview_p obi_new_view(OBIDMS_p dms, const_char_p view_name, Obiview_p view_to_clone, index_t* line_selection, const_char_p comments)
+    
+    Obiview_p obi_new_view_cloned_from_name(OBIDMS_p dms, const_char_p view_name, const_char_p view_to_clone_name, index_t* line_selection, const_char_p comments)
+
+    Obiview_p obi_new_view_nuc_seqs_cloned_from_name(OBIDMS_p dms, const_char_p view_name, const_char_p view_to_clone_name, index_t* line_selection, const_char_p comments)
+              
+    Obiview_infos_p obi_view_map_file(OBIDMS_p dms, const char* view_name)
+
+    int obi_view_unmap_file(OBIDMS_p dms, Obiview_infos_p view_infos)
+
+    Obiview_p obi_open_view(OBIDMS_p dms, const_char_p view_name)
+    
+    int obi_view_add_column(Obiview_p view,
+                            const_char_p column_name,
+                            obiversion_t version_number,
+                            OBIType_t data_type,
+                            index_t nb_lines,
+                            index_t nb_elements_per_line,
+                            const_char_p elements_names,
+                            const_char_p indexer_name,
+                            const_char_p comments,
+                            bint create)
+
+    int obi_view_delete_column(Obiview_p view, const_char_p column_name)
+    
+    int obi_select_line(Obiview_p view, index_t line_nb)
+
+    int obi_select_lines(Obiview_p view, index_t* line_nbs)
+        
+    OBIDMS_column_p obi_view_get_column(Obiview_p view, const_char_p column_name)
+
+    OBIDMS_column_p* obi_view_get_pointer_on_column_in_view(Obiview_p view, const_char_p column_name)
+
+    int obi_save_view(Obiview_p view)
+    
+    int obi_close_view(Obiview_p view)
+    
+    int obi_save_and_close_view(Obiview_p view)
+
+    int obi_column_set_obiint_with_elt_name_in_view(Obiview_p view,
+                                                    OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    const_char_p element_name, 
+                                                    obiint_t value)
+
+    int obi_column_set_obiint_with_elt_idx_in_view(Obiview_p view,
+                                                   OBIDMS_column_p column, 
+                                                   index_t line_nb, 
+                                                   index_t element_idx, 
+                                                   obiint_t value)
+    
+    obiint_t obi_column_get_obiint_with_elt_name_in_view(Obiview_p view,
+                                                         OBIDMS_column_p column, 
+                                                         index_t line_nb, 
+                                                         const_char_p element_name)
+    
+    obiint_t obi_column_get_obiint_with_elt_idx_in_view(Obiview_p view,
+                                                        OBIDMS_column_p column, 
+                                                        index_t line_nb, 
+                                                        index_t element_idx)
+
+    int obi_column_set_obibool_with_elt_name_in_view(Obiview_p view,
+                                                     OBIDMS_column_p column, 
+                                                     index_t line_nb, 
+                                                     const_char_p element_name, 
+                                                     obibool_t value)
+
+    int obi_column_set_obibool_with_elt_idx_in_view(Obiview_p view,
+                                                    OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    index_t element_idx, 
+                                                    obibool_t value)
+    
+    obibool_t obi_column_get_obibool_with_elt_name_in_view(Obiview_p view,
+                                                           OBIDMS_column_p column, 
+                                                           index_t line_nb, 
+                                                           const_char_p element_name)
+    
+    obibool_t obi_column_get_obibool_with_elt_idx_in_view(Obiview_p view,
+                                                          OBIDMS_column_p column, 
+                                                          index_t line_nb, 
+                                                          index_t element_idx)
+
+    int obi_column_set_obichar_with_elt_name_in_view(Obiview_p view,
+                                                     OBIDMS_column_p column, 
+                                                     index_t line_nb, 
+                                                     const_char_p element_name, 
+                                                     obichar_t value)
+
+    int obi_column_set_obichar_with_elt_idx_in_view(Obiview_p view,
+                                                    OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    index_t element_idx, 
+                                                    obichar_t value)
+    
+    obichar_t obi_column_get_obichar_with_elt_name_in_view(Obiview_p view,
+                                                           OBIDMS_column_p column, 
+                                                           index_t line_nb, 
+                                                           const_char_p element_name)
+    
+    obichar_t obi_column_get_obichar_with_elt_idx_in_view(Obiview_p view,
+                                                          OBIDMS_column_p column, 
+                                                          index_t line_nb, 
+                                                          index_t element_idx)
+
+    int obi_column_set_obifloat_with_elt_name_in_view(Obiview_p view,
+                                                      OBIDMS_column_p column, 
+                                                      index_t line_nb, 
+                                                      const_char_p element_name, 
+                                                      obifloat_t value)
+
+    int obi_column_set_obifloat_with_elt_idx_in_view(Obiview_p view,
+                                                     OBIDMS_column_p column, 
+                                                     index_t line_nb, 
+                                                     index_t element_idx, 
+                                                     obifloat_t value)
+    
+    obifloat_t obi_column_get_obifloat_with_elt_name_in_view(Obiview_p view,
+                                                             OBIDMS_column_p column, 
+                                                             index_t line_nb, 
+                                                             const_char_p element_name)
+    
+    obifloat_t obi_column_get_obifloat_with_elt_idx_in_view(Obiview_p view,
+                                                            OBIDMS_column_p column, 
+                                                            index_t line_nb, 
+                                                            index_t element_idx)
+
+    int obi_column_set_obiqual_char_with_elt_idx_in_view(Obiview_p view, 
+                                                         OBIDMS_column_p column, 
+                                                         index_t line_nb, 
+                                                         index_t element_idx, 
+                                                         const char* value)
+
+    int obi_column_set_obiqual_int_with_elt_idx_in_view(Obiview_p view, 
+                                                        OBIDMS_column_p column, 
+                                                        index_t line_nb, 
+                                                        index_t element_idx, 
+                                                        const uint8_t* value, 
+                                                        int value_length)
+
+    char* obi_column_get_obiqual_char_with_elt_idx_in_view(Obiview_p view, 
+                                                           OBIDMS_column_p column, 
+                                                           index_t line_nb, 
+                                                           index_t element_idx)
+
+    const uint8_t* obi_column_get_obiqual_int_with_elt_idx_in_view(Obiview_p view, 
+                                                                   OBIDMS_column_p column, 
+                                                                   index_t line_nb, 
+                                                                   index_t element_idx, 
+                                                                   int* value_length)
+
+    int obi_column_set_obiqual_char_with_elt_name_in_view(Obiview_p view, 
+                                                          OBIDMS_column_p column, 
+                                                          index_t line_nb, 
+                                                          const char* element_name, 
+                                                          const char* value)
+
+    int obi_column_set_obiqual_int_with_elt_name_in_view(Obiview_p view, 
+                                                         OBIDMS_column_p column, 
+                                                         index_t line_nb, 
+                                                         const char* element_name, 
+                                                         const uint8_t* value, 
+                                                         int value_length)
+
+    char* obi_column_get_obiqual_char_with_elt_name_in_view(Obiview_p view, 
+                                                            OBIDMS_column_p column, 
+                                                            index_t line_nb, 
+                                                            const char* element_name)
+
+    const uint8_t* obi_column_get_obiqual_int_with_elt_name_in_view(Obiview_p view, 
+                                                                    OBIDMS_column_p column, 
+                                                                    index_t line_nb, 
+                                                                    const char* element_name, 
+                                                                    int* value_length)
+
+    int obi_column_set_obistr_with_elt_name_in_view(Obiview_p view,
+                                                    OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    const_char_p element_name, 
+                                                    const_char_p value)
+
+    int obi_column_set_obistr_with_elt_idx_in_view(Obiview_p view,
+                                                   OBIDMS_column_p column, 
+                                                   index_t line_nb, 
+                                                   index_t element_idx, 
+                                                   const_char_p value)
+    
+    const_char_p obi_column_get_obistr_with_elt_name_in_view(Obiview_p view,
+                                                           OBIDMS_column_p column, 
+                                                           index_t line_nb, 
+                                                           const_char_p element_name)
+    
+    const_char_p obi_column_get_obistr_with_elt_idx_in_view(Obiview_p view,
+                                                            OBIDMS_column_p column, 
+                                                            index_t line_nb, 
+                                                            index_t element_idx)
+
+    int obi_column_set_obiseq_with_elt_name_in_view(Obiview_p view,
+                                                    OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    const_char_p element_name, 
+                                                    const_char_p value)
+
+    int obi_column_set_obiseq_with_elt_idx_in_view(Obiview_p view,
+                                                   OBIDMS_column_p column, 
+                                                   index_t line_nb, 
+                                                   index_t element_idx, 
+                                                   const_char_p value)
+    
+    char* obi_column_get_obiseq_with_elt_name_in_view(Obiview_p view,
+                                                      OBIDMS_column_p column, 
+                                                      index_t line_nb, 
+                                                      const_char_p element_name)
+    
+    char* obi_column_get_obiseq_with_elt_idx_in_view(Obiview_p view,
+                                                     OBIDMS_column_p column, 
+                                                     index_t line_nb, 
+                                                     index_t element_idx)
+

python/obitools3/obigrep.py

@@ -0,0 +1,51 @@
+import sys
+import argparse
+
+from obitools3.obidms._obidms import OBIDMS
+
+if __name__ == '__main__': 
+    
+    parser = argparse.ArgumentParser(description='Pseudo obigrep.')
+    
+    parser.add_argument('-V', '--view', dest='view', type=str,
+                       help='Name of the view that should be considered')    
+
+    parser.add_argument('-N', '--new_view', dest='new_view', type=str,
+                       help='Name of the new view that should be created')    
+    
+#     parser.add_argument('-k', '--key', dest='key', type=str,
+#                        help='Name of the key that should be considered')
+# 
+#     parser.add_argument('-c', '--comp', dest='comparison', type=int,
+#                        help='Comparison to be made: -1:< ; 0:== ; 1:>')
+# 
+#     parser.add_argument('-v', '--value', dest='value', type=object,
+#                        help='Value to be compared')
+
+    args = parser.parse_args()
+    
+    d = OBIDMS('tdms')
+    
+    #condition = 1
+    line_selec = []
+    
+    v = d.open_view(args.view)
+    
+    i = 0
+    for l in v :
+        if l['score'] > 350 : 
+            line_selec.append(i)
+        i+=1
+
+    new_v = d.new_view(args.new_view, view_to_clone=v, line_selection=line_selec, view_type="NUC_SEQS_VIEW", comments="obigrep "+args.view+" to "+args.new_view) #args.key+" "+str(args.comparison)+" "+str(args.value)+" "+)
+
+    print("\n")
+    print(new_v.__repr__())
+
+    v.save_and_close()
+    new_v.save_and_close()
+    
+    d.close()
+
+    print("\nDone.")
+    

python/obitools3/obihead.py

@@ -0,0 +1,43 @@
+import sys
+import argparse
+
+from obitools3.obidms._obidms import OBIDMS
+
+if __name__ == '__main__': 
+    
+    parser = argparse.ArgumentParser(description='Pseudo obihead.')
+    
+    parser.add_argument('-V', '--view', dest='view', type=str,
+                       help='Name of the view that should be considered')    
+
+    parser.add_argument('-N', '--new_view', dest='new_view', type=str,
+                       help='Name of the new view that should be created')    
+    
+    parser.add_argument('-n', '--nb', dest='nb_lines', type=int,
+                       help='Number of lines that should be taken')    
+    
+
+    args = parser.parse_args()
+    
+    d = OBIDMS('tdms')
+    
+    #condition = 1
+    line_selec = []
+    
+    v = d.open_view(args.view)
+    
+    for i in range(0, args.nb_lines) :
+        line_selec.append(i)
+        
+    new_v = d.new_view(args.new_view, view_to_clone=v, line_selection=line_selec, view_type="NUC_SEQS_VIEW", comments="obihead "+str(args.nb_lines)+", "+args.view+" to "+args.new_view) #args.key+" "+str(args.comparison)+" "+str(args.value)+" "+)
+
+    print("\n")
+    print(new_v.__repr__())
+
+    v.save_and_close()
+    new_v.save_and_close()
+    
+    d.close()
+
+    print("\nDone.")
+    

python/obitools3/obiimport.py

@@ -0,0 +1,199 @@
+import sys
+import argparse
+import time
+
+from obitools3.obidms._obidms import OBIDMS
+
+
+def bufferedRead(fileobj,size=209715200):   ## 200 MB
+   buffer = fileobj.readlines(size)
+   while buffer:
+       for l in buffer:
+           yield l
+       buffer = fileobj.readlines(size)
+
+
+if __name__ == '__main__': 
+    
+    parser = argparse.ArgumentParser(description='Convert a fasta file in an OBIDMS.')
+    
+    parser.add_argument('-i', '--input', dest='input_file', type=str,
+                       help='Name of the file containing the sequences')
+
+    args = parser.parse_args()
+
+    d = OBIDMS('tdms')
+    
+    view = d.new_view('uniq view', view_type="NUC_SEQS_VIEW")
+
+#    for i in range(35000000) :
+#        if (not (i%500000)) :
+#            print(str(time.time())+'\t'+str(i))
+#        id = "@HWI-D00405:142:C71BAANXX:4:1101:1234:2234_CONS_SUB_SUB_"+str(i)
+#        view[i].set_id(id)
+#        if id != view[i]["ID"] :
+#            print("nope", id, view[i]["ID"])
+    
+    input_file = open(args.input_file, 'r')
+    input_file_buffered = bufferedRead(input_file)
+    
+#         
+#     if args.input_file[-1:] == "a" :
+#     
+#         i = 0
+#         next = False
+#         first = True
+#         
+#         for line in input_file :
+#             
+#             if line[0] == ">" :
+#                 
+#                 if not first :
+#                     # save seq
+#                     #print(i, id, seq)
+#                     view[i].set_sequence(seq)
+#                     i+=1
+#                 
+#                 first = False
+#                     
+#                 #id = line.split(" ", 1)[0][1:]
+#                 #rest = (line[:-1].split(" ", 1)[1]).split(";")
+#                 #view[i].set_id(id)
+#                 
+# #                 description = ""
+# #                 for j in range(len(rest)) :
+# #                     if "=" in rest[j] :
+# #                         rest[j] = rest[j].strip()
+# #                         rest[j] = rest[j].split("=", 1)
+# #                         column_name = rest[j][0]
+# #                         v = rest[j][1]
+# #                         if ((not v.isalpha()) and (v.isalnum())) :
+# #                             conv_v = int(v)
+# #                         elif (v == "True") or (v == "False") :
+# #                             conv_v = bool(v)
+# #                         else :
+# #                             f = True
+# #                             for letter in v :
+# #                                 if ((not letter.isalnum()) or (letter != ".")) :
+# #                                     f = False
+# #                             if f :
+# #                                 conv_v = float(v)
+# #                             else : 
+# #                                 conv_v = v
+# #                         view[i][column_name] = conv_v
+# #                     else :
+# #                         description+=rest[j]
+# #                 
+# #                 if description != "" :
+# #                     description = description.strip()
+# #                     view[i].set_description(description)
+#                     
+#                 #print(id)
+#                 #print(rest)
+#                 #print(description)
+#                 
+#                 next = True
+#                 
+#             elif next == True :
+#                 
+#      #           if not (i % 1E5) :
+#      #               print(i)
+#                 
+#                 seq = line[:-1]
+#                 next = False
+#             
+#             elif not next :
+#                 
+#                 seq += line[:-1]
+# 
+# 
+#     elif args.input_file[-1:] == "q" :
+#         
+#         i = 0
+#         l = 0
+#         next = False
+#
+    l=0
+    i=0
+#     while (True):
+#         l+=1
+#         line = input_file.readline()
+#         if line=="":
+#             break
+    for line in input_file_buffered :
+# 
+             #if i > 1E7 :
+#             #    print('hmm?')
+#                         
+#        if i == 6000000 :
+#            break
+#             
+        if l%4 == 0 :
+#                
+            if (not (i%500000)) :
+                print(str(time.time())+'\t'+str(i))
+# # 
+# #                 #print("header", line)
+# # 
+            id = line.split(" ", 1)[0][1:]
+#            print(id)
+# #                 #rest = (line[:-1].split(" ", 1)[1]).split(";")
+            view[i].set_id(id)
+#            print(view[i]["ID"])
+# 
+#            i+=1
+        
+#        l+=1
+#                 
+# #                 description = ""
+# #                 for j in range(len(rest)) :
+# #                     if "=" in rest[j] :
+# #                         rest[j] = rest[j].strip()
+# #                         rest[j] = rest[j].split("=", 1)
+# #                         column_name = rest[j][0]
+# #                         #print("COLUMN", column_name)
+# #                         v = rest[j][1]
+# #                         if (v == "") and (column_name in view) and (view[column_name].get_data_type() == "OBI_SEQ") :
+# #                             #print(">>>>>>YUP")
+# #                             conv_v = "aa"
+# #                         else :
+# #                             if ((not v.isalpha()) and (v.isalnum())) :
+# #                                 conv_v = int(v)
+# #                             elif (v == "True") or (v == "False") :
+# #                                 conv_v = bool(v)
+# #                             else :
+# #                                 f = True
+# #                                 for letter in v :
+# #                                     if ((not letter.isalnum()) or (letter != ".")) :
+# #                                         f = False
+# #                                 if f :
+# #                                     conv_v = float(v)
+# #                                 else :
+# #                                     conv_v = v
+# #                         view[i][column_name] = conv_v
+# #                     else :
+# #                         description+=rest[j]
+# #                 
+# #                 if description != "" :
+# #                     description = description.strip()
+# #                     view[i].set_description(description)
+#                 
+        elif l%4 == 1 :
+# #                
+            seq = line[:-1]
+#                 #print("seq", seq)
+            view[i].set_sequence(seq)
+            i+=1
+#                         
+        l+=1
+#     
+#     
+    input_file.close()
+
+    #print(view)
+    print(view.__repr__())
+
+    view.save_and_close()
+    d.close()
+
+    print("Done.")

python/obitools3/parsers/fasta.pxd

@@ -0,0 +1,9 @@
+#cython: language_level=3
+
+from ..utils cimport str2bytes
+from .header cimport parseHeader
+from ..files.universalopener cimport uopen
+from ..files.linebuffer cimport LineBuffer
+
+
+