Pseudo obihead for tests

creating a column not using indexers

LICENSE

@@ -0,0 +1,518 @@
+
+  CeCILL FREE SOFTWARE LICENSE AGREEMENT
+
+Version 2.1 dated 2013-06-21
+
+
+    Notice
+
+This Agreement is a Free Software license agreement that is the result
+of discussions between its authors in order to ensure compliance with
+the two main principles guiding its drafting:
+
+  * firstly, compliance with the principles governing the distribution
+    of Free Software: access to source code, broad rights granted to users,
+  * secondly, the election of a governing law, French law, with which it
+    is conformant, both as regards the law of torts and intellectual
+    property law, and the protection that it offers to both authors and
+    holders of the economic rights over software.
+
+The authors of the CeCILL (for Ce[a] C[nrs] I[nria] L[ogiciel] L[ibre]) 
+license are: 
+
+Commissariat à l'énergie atomique et aux énergies alternatives - CEA, a
+public scientific, technical and industrial research establishment,
+having its principal place of business at 25 rue Leblanc, immeuble Le
+Ponant D, 75015 Paris, France.
+
+Centre National de la Recherche Scientifique - CNRS, a public scientific
+and technological establishment, having its principal place of business
+at 3 rue Michel-Ange, 75794 Paris cedex 16, France.
+
+Institut National de Recherche en Informatique et en Automatique -
+Inria, a public scientific and technological establishment, having its
+principal place of business at Domaine de Voluceau, Rocquencourt, BP
+105, 78153 Le Chesnay cedex, France.
+
+
+    Preamble
+
+The purpose of this Free Software license agreement is to grant users
+the right to modify and redistribute the software governed by this
+license within the framework of an open source distribution model.
+
+The exercising of this right is conditional upon certain obligations for
+users so as to preserve this status for all subsequent redistributions.
+
+In consideration of access to the source code and the rights to copy,
+modify and redistribute granted by the license, users are provided only
+with a limited warranty and the software's author, the holder of the
+economic rights, and the successive licensors only have limited liability.
+
+In this respect, the risks associated with loading, using, modifying
+and/or developing or reproducing the software by the user are brought to
+the user's attention, given its Free Software status, which may make it
+complicated to use, with the result that its use is reserved for
+developers and experienced professionals having in-depth computer
+knowledge. Users are therefore encouraged to load and test the
+suitability of the software as regards their requirements in conditions
+enabling the security of their systems and/or data to be ensured and,
+more generally, to use and operate it in the same conditions of
+security. This Agreement may be freely reproduced and published,
+provided it is not altered, and that no provisions are either added or
+removed herefrom.
+
+This Agreement may apply to any or all software for which the holder of
+the economic rights decides to submit the use thereof to its provisions.
+
+Frequently asked questions can be found on the official website of the
+CeCILL licenses family (http://www.cecill.info/index.en.html) for any 
+necessary clarification.
+
+
+    Article 1 - DEFINITIONS
+
+For the purpose of this Agreement, when the following expressions
+commence with a capital letter, they shall have the following meaning:
+
+Agreement: means this license agreement, and its possible subsequent
+versions and annexes.
+
+Software: means the software in its Object Code and/or Source Code form
+and, where applicable, its documentation, "as is" when the Licensee
+accepts the Agreement.
+
+Initial Software: means the Software in its Source Code and possibly its
+Object Code form and, where applicable, its documentation, "as is" when
+it is first distributed under the terms and conditions of the Agreement.
+
+Modified Software: means the Software modified by at least one
+Contribution.
+
+Source Code: means all the Software's instructions and program lines to
+which access is required so as to modify the Software.
+
+Object Code: means the binary files originating from the compilation of
+the Source Code.
+
+Holder: means the holder(s) of the economic rights over the Initial
+Software.
+
+Licensee: means the Software user(s) having accepted the Agreement.
+
+Contributor: means a Licensee having made at least one Contribution.
+
+Licensor: means the Holder, or any other individual or legal entity, who
+distributes the Software under the Agreement.
+
+Contribution: means any or all modifications, corrections, translations,
+adaptations and/or new functions integrated into the Software by any or
+all Contributors, as well as any or all Internal Modules.
+
+Module: means a set of sources files including their documentation that
+enables supplementary functions or services in addition to those offered
+by the Software.
+
+External Module: means any or all Modules, not derived from the
+Software, so that this Module and the Software run in separate address
+spaces, with one calling the other when they are run.
+
+Internal Module: means any or all Module, connected to the Software so
+that they both execute in the same address space.
+
+GNU GPL: means the GNU General Public License version 2 or any
+subsequent version, as published by the Free Software Foundation Inc.
+
+GNU Affero GPL: means the GNU Affero General Public License version 3 or
+any subsequent version, as published by the Free Software Foundation Inc.
+
+EUPL: means the European Union Public License version 1.1 or any
+subsequent version, as published by the European Commission.
+
+Parties: mean both the Licensee and the Licensor.
+
+These expressions may be used both in singular and plural form.
+
+
+    Article 2 - PURPOSE
+
+The purpose of the Agreement is the grant by the Licensor to the
+Licensee of a non-exclusive, transferable and worldwide license for the
+Software as set forth in Article 5 <#scope> hereinafter for the whole
+term of the protection granted by the rights over said Software.
+
+
+    Article 3 - ACCEPTANCE
+
+3.1 The Licensee shall be deemed as having accepted the terms and
+conditions of this Agreement upon the occurrence of the first of the
+following events:
+
+  * (i) loading the Software by any or all means, notably, by
+    downloading from a remote server, or by loading from a physical medium;
+  * (ii) the first time the Licensee exercises any of the rights granted
+    hereunder.
+
+3.2 One copy of the Agreement, containing a notice relating to the
+characteristics of the Software, to the limited warranty, and to the
+fact that its use is restricted to experienced users has been provided
+to the Licensee prior to its acceptance as set forth in Article 3.1
+<#accepting> hereinabove, and the Licensee hereby acknowledges that it
+has read and understood it.
+
+
+    Article 4 - EFFECTIVE DATE AND TERM
+
+
+      4.1 EFFECTIVE DATE
+
+The Agreement shall become effective on the date when it is accepted by
+the Licensee as set forth in Article 3.1 <#accepting>.
+
+
+      4.2 TERM
+
+The Agreement shall remain in force for the entire legal term of
+protection of the economic rights over the Software.
+
+
+    Article 5 - SCOPE OF RIGHTS GRANTED
+
+The Licensor hereby grants to the Licensee, who accepts, the following
+rights over the Software for any or all use, and for the term of the
+Agreement, on the basis of the terms and conditions set forth hereinafter.
+
+Besides, if the Licensor owns or comes to own one or more patents
+protecting all or part of the functions of the Software or of its
+components, the Licensor undertakes not to enforce the rights granted by
+these patents against successive Licensees using, exploiting or
+modifying the Software. If these patents are transferred, the Licensor
+undertakes to have the transferees subscribe to the obligations set
+forth in this paragraph.
+
+
+      5.1 RIGHT OF USE
+
+The Licensee is authorized to use the Software, without any limitation
+as to its fields of application, with it being hereinafter specified
+that this comprises:
+
+ 1. permanent or temporary reproduction of all or part of the Software
+    by any or all means and in any or all form.
+
+ 2. loading, displaying, running, or storing the Software on any or all
+    medium.
+
+ 3. entitlement to observe, study or test its operation so as to
+    determine the ideas and principles behind any or all constituent
+    elements of said Software. This shall apply when the Licensee
+    carries out any or all loading, displaying, running, transmission or
+    storage operation as regards the Software, that it is entitled to
+    carry out hereunder.
+
+
+      5.2 ENTITLEMENT TO MAKE CONTRIBUTIONS
+
+The right to make Contributions includes the right to translate, adapt,
+arrange, or make any or all modifications to the Software, and the right
+to reproduce the resulting software.
+
+The Licensee is authorized to make any or all Contributions to the
+Software provided that it includes an explicit notice that it is the
+author of said Contribution and indicates the date of the creation thereof.
+
+
+      5.3 RIGHT OF DISTRIBUTION
+
+In particular, the right of distribution includes the right to publish,
+transmit and communicate the Software to the general public on any or
+all medium, and by any or all means, and the right to market, either in
+consideration of a fee, or free of charge, one or more copies of the
+Software by any means.
+
+The Licensee is further authorized to distribute copies of the modified
+or unmodified Software to third parties according to the terms and
+conditions set forth hereinafter.
+
+
+        5.3.1 DISTRIBUTION OF SOFTWARE WITHOUT MODIFICATION
+
+The Licensee is authorized to distribute true copies of the Software in
+Source Code or Object Code form, provided that said distribution
+complies with all the provisions of the Agreement and is accompanied by:
+
+ 1. a copy of the Agreement,
+
+ 2. a notice relating to the limitation of both the Licensor's warranty
+    and liability as set forth in Articles 8 and 9,
+
+and that, in the event that only the Object Code of the Software is
+redistributed, the Licensee allows effective access to the full Source
+Code of the Software for a period of at least three years from the
+distribution of the Software, it being understood that the additional
+acquisition cost of the Source Code shall not exceed the cost of the
+data transfer.
+
+
+        5.3.2 DISTRIBUTION OF MODIFIED SOFTWARE
+
+When the Licensee makes a Contribution to the Software, the terms and
+conditions for the distribution of the resulting Modified Software
+become subject to all the provisions of this Agreement.
+
+The Licensee is authorized to distribute the Modified Software, in
+source code or object code form, provided that said distribution
+complies with all the provisions of the Agreement and is accompanied by:
+
+ 1. a copy of the Agreement,
+
+ 2. a notice relating to the limitation of both the Licensor's warranty
+    and liability as set forth in Articles 8 and 9,
+
+and, in the event that only the object code of the Modified Software is
+redistributed,
+
+ 3. a note stating the conditions of effective access to the full source
+    code of the Modified Software for a period of at least three years
+    from the distribution of the Modified Software, it being understood
+    that the additional acquisition cost of the source code shall not
+    exceed the cost of the data transfer.
+
+
+        5.3.3 DISTRIBUTION OF EXTERNAL MODULES
+
+When the Licensee has developed an External Module, the terms and
+conditions of this Agreement do not apply to said External Module, that
+may be distributed under a separate license agreement.
+
+
+        5.3.4 COMPATIBILITY WITH OTHER LICENSES
+
+The Licensee can include a code that is subject to the provisions of one
+of the versions of the GNU GPL, GNU Affero GPL and/or EUPL in the
+Modified or unmodified Software, and distribute that entire code under
+the terms of the same version of the GNU GPL, GNU Affero GPL and/or EUPL.
+
+The Licensee can include the Modified or unmodified Software in a code
+that is subject to the provisions of one of the versions of the GNU GPL,
+GNU Affero GPL and/or EUPL and distribute that entire code under the
+terms of the same version of the GNU GPL, GNU Affero GPL and/or EUPL.
+
+
+    Article 6 - INTELLECTUAL PROPERTY
+
+
+      6.1 OVER THE INITIAL SOFTWARE
+
+The Holder owns the economic rights over the Initial Software. Any or
+all use of the Initial Software is subject to compliance with the terms
+and conditions under which the Holder has elected to distribute its work
+and no one shall be entitled to modify the terms and conditions for the
+distribution of said Initial Software.
+
+The Holder undertakes that the Initial Software will remain ruled at
+least by this Agreement, for the duration set forth in Article 4.2 <#term>.
+
+
+      6.2 OVER THE CONTRIBUTIONS
+
+The Licensee who develops a Contribution is the owner of the
+intellectual property rights over this Contribution as defined by
+applicable law.
+
+
+      6.3 OVER THE EXTERNAL MODULES
+
+The Licensee who develops an External Module is the owner of the
+intellectual property rights over this External Module as defined by
+applicable law and is free to choose the type of agreement that shall
+govern its distribution.
+
+
+      6.4 JOINT PROVISIONS
+
+The Licensee expressly undertakes:
+
+ 1. not to remove, or modify, in any manner, the intellectual property
+    notices attached to the Software;
+
+ 2. to reproduce said notices, in an identical manner, in the copies of
+    the Software modified or not.
+
+The Licensee undertakes not to directly or indirectly infringe the
+intellectual property rights on the Software of the Holder and/or
+Contributors, and to take, where applicable, vis-à-vis its staff, any
+and all measures required to ensure respect of said intellectual
+property rights of the Holder and/or Contributors.
+
+
+    Article 7 - RELATED SERVICES
+
+7.1 Under no circumstances shall the Agreement oblige the Licensor to
+provide technical assistance or maintenance services for the Software.
+
+However, the Licensor is entitled to offer this type of services. The
+terms and conditions of such technical assistance, and/or such
+maintenance, shall be set forth in a separate instrument. Only the
+Licensor offering said maintenance and/or technical assistance services
+shall incur liability therefor.
+
+7.2 Similarly, any Licensor is entitled to offer to its licensees, under
+its sole responsibility, a warranty, that shall only be binding upon
+itself, for the redistribution of the Software and/or the Modified
+Software, under terms and conditions that it is free to decide. Said
+warranty, and the financial terms and conditions of its application,
+shall be subject of a separate instrument executed between the Licensor
+and the Licensee.
+
+
+    Article 8 - LIABILITY
+
+8.1 Subject to the provisions of Article 8.2, the Licensee shall be
+entitled to claim compensation for any direct loss it may have suffered
+from the Software as a result of a fault on the part of the relevant
+Licensor, subject to providing evidence thereof.
+
+8.2 The Licensor's liability is limited to the commitments made under
+this Agreement and shall not be incurred as a result of in particular:
+(i) loss due the Licensee's total or partial failure to fulfill its
+obligations, (ii) direct or consequential loss that is suffered by the
+Licensee due to the use or performance of the Software, and (iii) more
+generally, any consequential loss. In particular the Parties expressly
+agree that any or all pecuniary or business loss (i.e. loss of data,
+loss of profits, operating loss, loss of customers or orders,
+opportunity cost, any disturbance to business activities) or any or all
+legal proceedings instituted against the Licensee by a third party,
+shall constitute consequential loss and shall not provide entitlement to
+any or all compensation from the Licensor.
+
+
+    Article 9 - WARRANTY
+
+9.1 The Licensee acknowledges that the scientific and technical
+state-of-the-art when the Software was distributed did not enable all
+possible uses to be tested and verified, nor for the presence of
+possible defects to be detected. In this respect, the Licensee's
+attention has been drawn to the risks associated with loading, using,
+modifying and/or developing and reproducing the Software which are
+reserved for experienced users.
+
+The Licensee shall be responsible for verifying, by any or all means,
+the suitability of the product for its requirements, its good working
+order, and for ensuring that it shall not cause damage to either persons
+or properties.
+
+9.2 The Licensor hereby represents, in good faith, that it is entitled
+to grant all the rights over the Software (including in particular the
+rights set forth in Article 5 <#scope>).
+
+9.3 The Licensee acknowledges that the Software is supplied "as is" by
+the Licensor without any other express or tacit warranty, other than
+that provided for in Article 9.2 <#good-faith> and, in particular,
+without any warranty as to its commercial value, its secured, safe,
+innovative or relevant nature.
+
+Specifically, the Licensor does not warrant that the Software is free
+from any error, that it will operate without interruption, that it will
+be compatible with the Licensee's own equipment and software
+configuration, nor that it will meet the Licensee's requirements.
+
+9.4 The Licensor does not either expressly or tacitly warrant that the
+Software does not infringe any third party intellectual property right
+relating to a patent, software or any other property right. Therefore,
+the Licensor disclaims any and all liability towards the Licensee
+arising out of any or all proceedings for infringement that may be
+instituted in respect of the use, modification and redistribution of the
+Software. Nevertheless, should such proceedings be instituted against
+the Licensee, the Licensor shall provide it with technical and legal
+expertise for its defense. Such technical and legal expertise shall be
+decided on a case-by-case basis between the relevant Licensor and the
+Licensee pursuant to a memorandum of understanding. The Licensor
+disclaims any and all liability as regards the Licensee's use of the
+name of the Software. No warranty is given as regards the existence of
+prior rights over the name of the Software or as regards the existence
+of a trademark.
+
+
+    Article 10 - TERMINATION
+
+10.1 In the event of a breach by the Licensee of its obligations
+hereunder, the Licensor may automatically terminate this Agreement
+thirty (30) days after notice has been sent to the Licensee and has
+remained ineffective.
+
+10.2 A Licensee whose Agreement is terminated shall no longer be
+authorized to use, modify or distribute the Software. However, any
+licenses that it may have granted prior to termination of the Agreement
+shall remain valid subject to their having been granted in compliance
+with the terms and conditions hereof.
+
+
+    Article 11 - MISCELLANEOUS
+
+
+      11.1 EXCUSABLE EVENTS
+
+Neither Party shall be liable for any or all delay, or failure to
+perform the Agreement, that may be attributable to an event of force
+majeure, an act of God or an outside cause, such as defective
+functioning or interruptions of the electricity or telecommunications
+networks, network paralysis following a virus attack, intervention by
+government authorities, natural disasters, water damage, earthquakes,
+fire, explosions, strikes and labor unrest, war, etc.
+
+11.2 Any failure by either Party, on one or more occasions, to invoke
+one or more of the provisions hereof, shall under no circumstances be
+interpreted as being a waiver by the interested Party of its right to
+invoke said provision(s) subsequently.
+
+11.3 The Agreement cancels and replaces any or all previous agreements,
+whether written or oral, between the Parties and having the same
+purpose, and constitutes the entirety of the agreement between said
+Parties concerning said purpose. No supplement or modification to the
+terms and conditions hereof shall be effective as between the Parties
+unless it is made in writing and signed by their duly authorized
+representatives.
+
+11.4 In the event that one or more of the provisions hereof were to
+conflict with a current or future applicable act or legislative text,
+said act or legislative text shall prevail, and the Parties shall make
+the necessary amendments so as to comply with said act or legislative
+text. All other provisions shall remain effective. Similarly, invalidity
+of a provision of the Agreement, for any reason whatsoever, shall not
+cause the Agreement as a whole to be invalid.
+
+
+      11.5 LANGUAGE
+
+The Agreement is drafted in both French and English and both versions
+are deemed authentic.
+
+
+    Article 12 - NEW VERSIONS OF THE AGREEMENT
+
+12.1 Any person is authorized to duplicate and distribute copies of this
+Agreement.
+
+12.2 So as to ensure coherence, the wording of this Agreement is
+protected and may only be modified by the authors of the License, who
+reserve the right to periodically publish updates or new versions of the
+Agreement, each with a separate number. These subsequent versions may
+address new issues encountered by Free Software.
+
+12.3 Any Software distributed under a given version of the Agreement may
+only be subsequently distributed under the same version of the Agreement
+or a subsequent version, subject to the provisions of Article 5.3.4
+<#compatibility>.
+
+
+    Article 13 - GOVERNING LAW AND JURISDICTION
+
+13.1 The Agreement is governed by French law. The Parties agree to
+endeavor to seek an amicable solution to any disagreements or disputes
+that may arise during the performance of the Agreement.
+
+13.2 Failing an amicable solution within two (2) months as from their
+occurrence, and unless emergency proceedings are necessary, the
+disagreements or disputes shall be referred to the Paris Courts having
+jurisdiction, by the more diligent Party.

distutils.ext/obidistutils/command/build_ext.py

@@ -7,107 +7,109 @@ Created on 13 fevr. 2014
 from distutils import log
 import os
 
-from Cython.Distutils import build_ext  as ori_build_ext  # @UnresolvedImport
-
-from Cython.Compiler import Options as cython_options  # @UnresolvedImport
 
 from distutils.errors import DistutilsSetupError
 
-class build_ext(ori_build_ext):
-
+try:
+    from Cython.Distutils import build_ext  as ori_build_ext  # @UnresolvedImport
+    from Cython.Compiler import Options as cython_options  # @UnresolvedImport
+    class build_ext(ori_build_ext):
     
-    def modifyDocScripts(self):
-        build_dir_file=open("doc/sphinx/build_dir.txt","w")
-        print(self.build_lib,file=build_dir_file)
-        build_dir_file.close()
         
-    def initialize_options(self):
-        ori_build_ext.initialize_options(self)  # @UndefinedVariable
-        self.littlebigman = None
-        self.built_files = None
-
-    
-    def finalize_options(self):
-        ori_build_ext.finalize_options(self)  # @UndefinedVariable
-
-        self.set_undefined_options('littlebigman',
-                                   ('littlebigman',  'littlebigman'))
-        
-        self.set_undefined_options('build_files',
-                                   ('files',  'built_files'))
-
-        self.cython_c_in_temp = 1
-                   
-        if self.littlebigman =='-DLITTLE_END':
-            if self.define is None:
-                self.define=[('LITTLE_END',None)]
-            else:
-                self.define.append('LITTLE_END',None)
-        
-    def substitute_sources(self,exe_name,sources):
-        """
-        Substitutes source file name starting by an @ by the actual
-        name of the built file (see --> build_files)
-        """
-        sources = list(sources)
-        for i in range(len(sources)):
-            message = "%s :-> %s" % (exe_name,sources[i])
-            if sources[i][0]=='@':
-                try:
-                    filename = self.built_files[sources[i][1:]]
-                except KeyError:
-                    tmpfilename = os.path.join(self.build_temp,sources[i][1:])
-                    if os.path.isfile       (tmpfilename):
-                        filename = tmpfilename
-                    else:
-                        raise DistutilsSetupError(
-                             'The %s filename declared in the source '
-                             'files of the program %s have not been '
-                             'built by the installation process' % (sources[i],
-                                                                     exe_name))
-                sources[i]=filename
-                log.info("%s changed to %s",message,filename)
-            else:
-                log.info("%s ok",message)
-
-        return sources
-
-    def build_extensions(self):
-        # First, sanity-check the 'extensions' list
-        
-        for ext in self.extensions:
-            ext.sources = self.substitute_sources(ext.name,ext.sources)
+        def modifyDocScripts(self):
+            build_dir_file=open("doc/sphinx/build_dir.txt","w")
+            print(self.build_lib,file=build_dir_file)
+            build_dir_file.close()
             
-        self.check_extensions_list(self.extensions)
-
-        for ext in self.extensions:
-            log.info("%s :-> %s",ext.name,ext.sources)
-            ext.sources = self.cython_sources(ext.sources, ext)
-            self.build_extension(ext)
-
-        
-    def run(self):
-        self.modifyDocScripts()
-
-        for cmd_name in self.get_sub_commands():
-            self.run_command(cmd_name)
-
-        cython_options.annotate = True
-        ori_build_ext.run(self)  # @UndefinedVariable
-        
-
-    def has_files(self):
-        return self.distribution.has_files()
-
-    def has_executables(self):
-        return self.distribution.has_executables()
+        def initialize_options(self):
+            ori_build_ext.initialize_options(self)  # @UndefinedVariable
+            self.littlebigman = None
+            self.built_files = None
     
-    sub_commands = [('build_files',has_files),
-                    ('build_cexe', has_executables)
-                    ] + \
-                   ori_build_ext.sub_commands
-
         
+        def finalize_options(self):
+            ori_build_ext.finalize_options(self)  # @UndefinedVariable
+    
+            self.set_undefined_options('littlebigman',
+                                       ('littlebigman',  'littlebigman'))
+            
+            self.set_undefined_options('build_files',
+                                       ('files',  'built_files'))
+    
+            self.cython_c_in_temp = 1
+                       
+            if self.littlebigman =='-DLITTLE_END':
+                if self.define is None:
+                    self.define=[('LITTLE_END',None)]
+                else:
+                    self.define.append('LITTLE_END',None)
+            
+        def substitute_sources(self,exe_name,sources):
+            """
+            Substitutes source file name starting by an @ by the actual
+            name of the built file (see --> build_files)
+            """
+            sources = list(sources)
+            for i in range(len(sources)):
+                message = "%s :-> %s" % (exe_name,sources[i])
+                if sources[i][0]=='@':
+                    try:
+                        filename = self.built_files[sources[i][1:]]
+                    except KeyError:
+                        tmpfilename = os.path.join(self.build_temp,sources[i][1:])
+                        if os.path.isfile       (tmpfilename):
+                            filename = tmpfilename
+                        else:
+                            raise DistutilsSetupError(
+                                 'The %s filename declared in the source '
+                                 'files of the program %s have not been '
+                                 'built by the installation process' % (sources[i],
+                                                                         exe_name))
+                    sources[i]=filename
+                    log.info("%s changed to %s",message,filename)
+                else:
+                    log.info("%s ok",message)
+    
+            return sources
+    
+        def build_extensions(self):
+            # First, sanity-check the 'extensions' list
+            
+            for ext in self.extensions:
+                ext.sources = self.substitute_sources(ext.name,ext.sources)
+                
+            self.check_extensions_list(self.extensions)
+    
+            for ext in self.extensions:
+                log.info("%s :-> %s",ext.name,ext.sources)
+                ext.sources = self.cython_sources(ext.sources, ext)
+                self.build_extension(ext)
+    
+            
+        def run(self):
+            self.modifyDocScripts()
+    
+            for cmd_name in self.get_sub_commands():
+                self.run_command(cmd_name)
+    
+            cython_options.annotate = True
+            ori_build_ext.run(self)  # @UndefinedVariable
+            
+    
+        def has_files(self):
+            return self.distribution.has_files()
+    
+        def has_executables(self):
+            return self.distribution.has_executables()
+        
+        sub_commands = [('build_files',has_files),
+                        ('build_cexe', has_executables)
+                        ] + \
+                       ori_build_ext.sub_commands
+
+except ImportError:
+    from distutils.command import build_ext  # @UnusedImport
+           
 
 
         

distutils.ext/obidistutils/serenity/bootstrappip.py

@@ -0,0 +1,36 @@
+'''
+Created on 22 janv. 2016
+
+@author: coissac
+'''
+
+import sys
+from urllib import request
+import os.path
+
+from obidistutils.serenity.util import get_serenity_dir
+from obidistutils.serenity.rerun import rerun_with_anothe_python
+from obidistutils.serenity.checkpython import is_a_virtualenv_python
+
+getpipurl="https://bootstrap.pypa.io/get-pip.py"
+
+def bootstrap():
+
+    getpipfile=os.path.join(get_serenity_dir(),"get-pip.py")
+    
+    with request.urlopen(getpipurl) as getpip:
+        with open(getpipfile,"wb") as out:
+            for l in getpip:
+                out.write(l)
+                
+    python = sys.executable
+    
+    if is_a_virtualenv_python():
+        command= "%s %s" % (python,getpipfile)        
+    else:
+        command= "%s %s --user" % (python,getpipfile)
+    
+    os.system(command)
+    
+    rerun_with_anothe_python(python)
+    

distutils.ext/obidistutils/serenity/checkpackage.py

@@ -7,8 +7,13 @@ Created on 2 oct. 2014
 import re
 import os
 
-import pip                                              # @UnresolvedImport
-from pip.utils import get_installed_distributions       # @UnresolvedImport
+try:
+    import pip                                              # @UnresolvedImport
+    from pip.utils import get_installed_distributions       # @UnresolvedImport
+except ImportError:
+    from .bootstrappip import bootstrap
+    bootstrap()
+
 from distutils.version import StrictVersion             # @UnusedImport
 from distutils.errors import DistutilsError
 from distutils import log

distutils.ext/obidistutils/serenity/virtual.py

@@ -59,7 +59,7 @@ def serenity_virtualenv(envname,package,version,minversion='3.4',maxversion=None
                     clear=True, 
                     symlinks=False, 
                     with_pip=True)
-        
+                
         # check the newly created virtualenv
         return serenity_virtualenv(envname,package,version)
     

doc/.gitignore

@@ -1,3 +1,5 @@
 /build/
 /doxygen/
 /build_dir.txt
+/.DS_Store
+/.gitignore

doc/Makefile

@@ -57,7 +57,7 @@ html:
 	@echo "Generating Doxygen documentation..."
 	doxygen Doxyfile
 	@echo "Doxygen documentation generated. \n"
-	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
+	$(SPHINXBUILD) -b html -c ./ $(ALLSPHINXOPTS) $(BUILDDIR)/html
 	@echo
 	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
 

doc/conf.py

@@ -36,7 +36,7 @@ extensions = [
     'sphinx.ext.pngmath',
     'sphinx.ext.ifconfig',
     'sphinx.ext.viewcode',
-    'breathe',
+#    'breathe',
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -51,7 +51,7 @@ source_suffix = '.rst'
 #source_encoding = 'utf-8-sig'
 
 # The master toctree document.
-master_doc = 'index'
+master_doc = 'source/index'
 
 # General information about the project.
 project = u'OBITools3'
@@ -292,7 +292,7 @@ texinfo_documents = [
 #texinfo_no_detailmenu = False
 
 #Breathe configuration
-sys.path.append( "../breathe/" )
-breathe_projects = { "OBITools3": "../doxygen/xml/" }
+sys.path.append( "breathe/" )
+breathe_projects = { "OBITools3": "doxygen/xml/" }
 breathe_default_project = "OBITools3"
 

doc/doxygen/xml/.gitignore

@@ -1,4 +0,0 @@
-# Ignore everything in this directory
-*
-# Except this file
-!.gitignore

doc/source/DMS.rst

@@ -13,7 +13,7 @@ Up to now, each of these categories of data were stored in separate
 files, and nothing made it mandatory to keep them together.
 
 
-The `Data Management System` (DMS) of OBITools3 can be regarded as a basic 
+The `Data Management System` (DMS) of OBITools3 can be viewed like a basic 
 database system.
 
 
@@ -27,9 +27,7 @@ OBIDMS UML
 
 An OBIDMS directory contains :
   * one `OBIDMS history file <#obidms-history-files>`_
-  * Two different kinds of directories :
-     * OBIDMS column directories
-     * OBIDMS column group directories containing OBIDMS column directories
+  * OBIDMS column directories
 
 
 OBIDMS column directories
@@ -39,16 +37,9 @@ OBIDMS column directories contain :
     * all the different versions of one OBIDMS column, under the form of different files (`OBIDMS column files <#obidms-column-files>`_)
     * one `OBIDMS version file <#obidms-version-files>`_
 
-The directory name is the column attribute, or sub-attribute if the column directory is in a column group directory.
+The directory name is the column attribute with the extension ``.obicol``.
 
-
-OBIDMS column group directories
-===============================
-
-OBIDMS column group directories contain OBIDMS column directories. They are used to store dictionary-like data, where 
-each key corresponds to an OBIDMS column.
-
-The directory name is the dictionary attribute. Each key is considered a sub-attribute and is associated to its column.
+Example: ``count.obicol``
 
 
 OBIDMS column files
@@ -57,7 +48,7 @@ OBIDMS column files
 Each OBIDMS column file contains :
 	* a header of a size equal to a multiple of PAGESIZE (PAGESIZE being equal to 4096 bytes
 	  on most systems) containing metadata
-	* one column of data with the same `OBIType <types.html#obitypes>`_
+	* Lines of data with the same `OBIType <types.html#obitypes>`_
 
 
 Header
@@ -79,7 +70,14 @@ The header of an OBIDMS column contains :
 Data
 ----
 
-A column of data with the same `OBIType <types.html#obitypes>`_.
+A line of data corresponds to a vector of elements. Each element is associated with an element name. 
+Elements names are stored in the header. The correspondance between an element and its name is done 
+using their order in the lists of elements and elements names.  This structure allows the storage of 
+dictionary-like data.
+
+Example: In the header, the attribute ``elements_names`` will be associated with the value ``"sample_1;
+sample_2;sample_3"``, and a line of data with the type ``OBInt_t`` will be stored as an ``OBInt_t`` vector 
+of size three e.g. ``5|8|4``.
 
 
 Mandatory columns
@@ -158,3 +156,5 @@ operations ever done in the OBIDMS directory and the views in between them :
 .. image:: ./images/history.png
     :width: 150 px
     :align: center
+
+

doc/source/data.rst

@@ -2,8 +2,8 @@
 Data in OBITools3
 #################
 
-The OBITools3 inaugure a new way to manage DNA metabarcoding data. 
-They rely on a `Data management System` (DMS) that can be considered as
+The OBITools3 introduce a new way to manage DNA metabarcoding data. 
+They rely on a `Data management System` (DMS) that can be viewed like 
 a simplified database system.
 
 

doc/source/guidelines.rst

@@ -70,7 +70,7 @@ Tickets should always be labeled with the branches for which they are relevant.
 Documentation
 *************
 
-C functions are documented in the header files.
+C functions are documented in the header files for public functions, and in the source file for private functions.
 
 
 **************
@@ -92,7 +92,7 @@ C99 :
 	* Object layer
 	* OBITools3 library
 
-`Python 3 <https://www.python.org/>`_ :
+`Python 3.5 <https://www.python.org/>`_ :
 	* Top layer code (scripts)	
 	
 For the documentation, `Sphinx <http://sphinx-doc.org/>`_ should be used for both the original 
@@ -111,6 +111,8 @@ Enum members, macros, constants:                ``ALL_CAPS``
 
 Functions, local variables:                     ``lower_case``
 
+Public functions:                               ``obi_lower_case``
+
 Functions that shouldn't be called directly:    ``_lower_case`` (``_`` prefix)
 
 Global variables:                                ``g_lower_case`` (``g_`` prefix)
@@ -120,9 +122,6 @@ Pointers:                                        ``pointer_ptr`` (``_ptr`` suffi
 .. note::
     Underscores are used to delimit 'words'.
 
-.. todo::
-    ``obi_function`` for public functions names?
-
 
 *****************
 Programming rules

doc/source/index.rst

@@ -11,7 +11,6 @@ OBITools3 documentation
 
    Programming guidelines <guidelines>
    Data structures <data>
-   Pistes de reflexion <pistes>
 
 
 Indices and tables

doc/source/specialvalues.rst

@@ -7,13 +7,16 @@ NA values
 =========
 
 All OBITypes have an associated NA (Not Available) value. 
-NA values are implemented by specifying an explicit NA value for each type, corresponding to the R standards:
+NA values are implemented by specifying an explicit NA value for each type, 
+corresponding to the R standards as much as possible:
 
-* For the types ``OBIInt_t``, ``OBIBool_t``, ``OBIIdx_t`` and ``OBITaxid_t``, the NA value is ``INT_MIN``.
+* For the type ``OBIInt_t``, the NA value is ``INT_MIN``.
 
-* For the type ``OBIChar_t``: the NA value is ``\0`` (?).
+* For the type ``OBIBool_t``, the NA value is ``2``.
 
-* For the type ``OBIStr_t`` : the NA value is ``\0`` (?).
+* For the type ``OBIIdx_t`` and ``OBITaxid_t``, the NA value is ``SIZE_MAX``.
+
+* For the type ``OBIChar_t``: the NA value is ``\0``.
 
 * For the type ``OBIFloat_t``::
 
@@ -29,7 +32,7 @@ NA values are implemented by specifying an explicit NA value for each type, corr
         x.word[hw] = 0x7ff00000;
         x.word[lw] = 1954;
         return x.value;
-    }   
+    }
 
 
 Minimum and maximum values for ``OBIInt_t``

doc/source/types.rst

@@ -7,11 +7,14 @@ OBITypes
 :download:`html version of the OBITypes UML file <UML/OBITypes_UML.class.violet.html>`
 
 
+.. image:: ./UML/Obicolumn_classes_UML.png
+
+:download:`html version of the OBIDMS classes UML file <UML/Obicolumn_classes_UML.class.violet.html>`
+
+
 .. toctree::
    :maxdepth: 2
 
    The elementary types <elementary>
    The containers <containers>
    Special values <specialvalues>
-
-

doc/sphinx/build_dir.txt

@@ -0,0 +1 @@
+build/lib.macosx-10.6-intel-3.5

doc/sphinx/source/pistes.rst

@@ -1,23 +0,0 @@
-###################
-Pistes de reflexion
-###################
-
-
-******************************
-Ce que l'on veut pouvoir faire
-******************************
-
-    * Gerer les valeurs manquantes
-    * Modifier une colonne en cours d'ecriture (mmap)
-    * Ajouter des valeurs a la fin du fichier d'une colonne en cours d'ecriture (mmap)
-    
-
-******
-Divers
-******
-
-* Si l'ordre d'une colonne est change, elle est reecrite (pas d'index).
-* Utilisation de semaphores pour la lecture
-* Utilisation de tas pour l'indexation des chaines de caracteres. Chaque colonne dont 
-  le type est OBIStr_t est stockee dans 3 fichiers : un fichier contenant les chaines, un 
-  fichier contenant les index, et un fichier contenant le tas.

python/obi.py

@@ -0,0 +1,50 @@
+#!/usr/local/bin/python3.4
+'''
+obi -- shortdesc
+
+obi is a description
+
+It defines classes_and_methods
+
+@author:     user_name
+
+@copyright:  2014 organization_name. All rights reserved.
+
+@license:    license
+
+@contact:    user_email
+@deffield    updated: Updated
+'''
+
+
+default_config = { 'software'       : "The OBITools",
+                   'log'            : False,
+                   'loglevel'       : 'INFO',
+                   'progress'       : True,
+                   'defaultdms'     : None
+                  }
+
+root_config_name='obi'
+
+from obitools3.apps.config import getConfiguration     # @UnresolvedImport
+from obitools3.version import version
+
+__all__     = []
+__version__ = version
+__date__    = '2014-09-28'
+__updated__ = '2014-09-28'
+
+DEBUG = 1
+TESTRUN = 0
+PROFILE = 0
+
+
+if __name__ =="__main__":
+    
+    config = getConfiguration(root_config_name,
+                              default_config)    
+                
+    config[root_config_name]['module'].run(config)
+    
+    
+    

python/obitools3/apps/arguments.pxd

@@ -0,0 +1,3 @@
+#cython: language_level=3
+
+cpdef buildArgumentParser(str configname, str softname)

python/obitools3/apps/arguments.pyx

@@ -0,0 +1,61 @@
+#cython: language_level=3
+
+'''
+Created on 27 mars 2016
+
+@author: coissac
+'''
+
+import argparse
+import sys
+
+
+from .command import getCommandsList
+
+class ObiParser(argparse.ArgumentParser): 
+    def error(self, message):
+        sys.stderr.write('error: %s\n' % message)
+        self.print_help()
+        sys.exit(2)
+
+cpdef buildArgumentParser(str configname, 
+                          str softname):
+    parser = ObiParser()
+    
+    parser.add_argument('--version',   dest='%s:version' % configname, 
+                                       action='store_true', 
+                                       default=False, 
+                        help='Print the version of %s' % softname)
+
+    parser.add_argument('--log',       dest='%s:log' % configname, 
+                                       action='store',
+                                       type=str,
+                                       default=None, 
+                        help='Create a logfile')
+
+    parser.add_argument('--no-progress', dest='%s:progress' % configname, 
+                                       action='store_false', 
+                                       default=None, 
+                        help='Do not print the progress bar during analyzes')
+
+    subparsers = parser.add_subparsers(title='subcommands',
+                                       description='valid subcommands',
+                                       help='additional help')
+    
+    commands = getCommandsList()
+    
+    for c in commands:
+        module = commands[c]
+        
+        if hasattr(module, "run"):
+            if hasattr(module, "__title__"):
+                sub = subparsers.add_parser(c,help=module.__title__)
+            else:
+                sub = subparsers.add_parser(c)
+    
+            if hasattr(module, "addOptions"):
+                module.addOptions(sub)
+            
+            sub.set_defaults(**{'%s:module'  % configname : module})
+                        
+    return parser

python/obitools3/apps/command.pxd

@@ -0,0 +1,3 @@
+#cython: language_level=3
+
+cdef object loadCommand(str name,loader)

python/obitools3/apps/command.pyx

@@ -0,0 +1,44 @@
+#cython: language_level=3
+
+'''
+Created on 27 mars 2016
+
+@author: coissac
+'''
+
+import pkgutil
+
+from obitools3 import commands
+
+cdef object loadCommand(str name,loader):
+    '''
+    Load a command module from its name and an ImpLoader
+    
+    This function is for internal use
+    
+    @param name:   name of the module
+    @type name: str 
+    @param loader: the module loader
+    @type loader: ImpLoader
+    
+    @return the loaded module
+    @rtype: module 
+    '''
+    
+    module = loader.find_module(name).load_module(name)
+    return module
+
+def getCommandsList():
+    '''
+    Returns the list of sub-commands available to the main `obi` command
+    
+    @return: a dict instance with key corresponding to each command and
+             value corresponding to the module
+             
+    @rtype: dict
+    '''
+    
+    cdef dict cmds = dict((x[1],loadCommand(x[1],x[0])) 
+                           for x in pkgutil.iter_modules(commands.__path__) 
+                           if not x[2])
+    return cmds

python/obitools3/apps/config.pxd

@@ -0,0 +1,10 @@
+#cython: language_level=3
+
+cpdef str setRootConfigName(str rootname)
+cpdef str getRootConfigName()
+
+cdef dict buildDefaultConfiguration(str root_config_name,
+                                    dict  config)
+
+cpdef dict getConfiguration(str root_config_name=?,
+                                    dict  config=?)

python/obitools3/apps/config.pyx

@@ -0,0 +1,103 @@
+#cython: language_level=3
+
+'''
+Created on 27 mars 2016
+
+@author: coissac
+'''
+
+import sys
+
+from .command   import  getCommandsList
+from .logging   cimport getLogger
+from .arguments cimport buildArgumentParser
+
+from ..version import version
+from _curses import version
+
+cdef dict __default_config__ = {}
+
+
+cpdef str setRootConfigName(str rootname):
+    global __default_config__
+    if '__root_config__' in __default_config__:
+        if __default_config__["__root_config__"] in __default_config__:
+            __default_config__[rootname]=__default_config__[__default_config__["__root_config__"]]
+            del __default_config__[__default_config__["__root_config__"]]
+    __default_config__['__root_config__']=rootname
+    return rootname
+
+cpdef str getRootConfigName():
+    global __default_config__
+    return __default_config__.get('__root_config__',None)
+
+cdef dict buildDefaultConfiguration(str root_config_name,
+                                    dict  config):
+    global __default_config__
+    
+    __default_config__.clear()
+    setRootConfigName(root_config_name)    
+
+    __default_config__[root_config_name]=config
+    
+    config['version']=version
+    
+    commands = getCommandsList()
+    
+    for c in commands:
+        module = commands[c]
+    
+        assert hasattr(module, "run")
+        
+        if hasattr(module, 'default_config'):
+            __default_config__[c]=module.default_config
+        else:
+            __default_config__[c]={}
+                        
+    return __default_config__
+        
+
+cpdef dict getConfiguration(str root_config_name="__default__",
+                            dict  config={}):
+    global __default_config__
+
+    if '__done__' in __default_config__:
+        return __default_config__
+    
+    if root_config_name=="__default__":
+        raise RuntimeError("No root_config_name specified")
+    
+    if not config:
+        raise RuntimeError("Base configuration is empty")
+    
+    
+    
+    config =  buildDefaultConfiguration(root_config_name,
+                                        config)
+    
+    parser = buildArgumentParser(root_config_name,
+                                 config[root_config_name]['software'])
+    
+    options = vars(parser.parse_args())
+
+    if options['%s:version' % root_config_name]:
+        print("%s - Version %s" % (config[root_config_name]['software'],
+                                   config[root_config_name]['version']))
+        sys.exit(0)
+    
+    for k in options:
+        section,key = k.split(':')
+        s = config[section]
+        if options[k] is not None:
+            s[key]=options[k]
+            
+    if not 'module' in config[root_config_name]:
+        print('\nError: No command specified',file=sys.stderr)
+        parser.print_help()
+        sys.exit(2)
+                
+    getLogger(config)
+    
+    config['__done__']=True
+            
+    return config

python/obitools3/apps/logging.pxd

@@ -0,0 +1,3 @@
+#cython: language_level=3
+
+cpdef getLogger(dict config)

python/obitools3/apps/logging.pyx

@@ -0,0 +1,46 @@
+#cython: language_level=3
+
+'''
+Created on 27 mars 2016
+
+@author: coissac
+'''
+
+import logging
+import sys
+
+cpdef getLogger(dict config):
+    '''
+    Returns the logger as defined by the command line option
+    or by the config file
+    :param config:
+    '''
+    
+    root = config["__root_config__"]
+ 
+    level  = config[root]['loglevel'] 
+    logfile= config[root]['log'] 
+    
+    rootlogger   = logging.getLogger()
+    logFormatter = logging.Formatter("%(asctime)s [%(levelname)-5.5s]  %(message)s")
+
+    stderrHandler = logging.StreamHandler(sys.stderr)
+    stderrHandler.setFormatter(logFormatter)
+
+    rootlogger.addHandler(stderrHandler)
+    
+    if logfile:
+        fileHandler = logging.FileHandler(logfile)
+        fileHandler.setFormatter(logFormatter)
+        rootlogger.addHandler(fileHandler)
+    
+    try:
+        loglevel = getattr(logging, level) 
+    except:
+        loglevel = logging.INFO
+        
+    rootlogger.setLevel(loglevel)
+    
+    config[root]['logger']=rootlogger
+    
+    return rootlogger

python/obitools3/apps/progress.pxd

@@ -0,0 +1,65 @@
+#cython: language_level=3
+
+from ..utils cimport str2bytes
+
+cdef extern from "stdio.h":
+    struct FILE
+    int fprintf(FILE *stream, char *format, ...)
+    FILE* stderr
+    ctypedef unsigned int off_t "unsigned long long"
+    
+cdef extern from "unistd.h":
+    int fsync(int fd);
+
+cdef extern from "time.h":
+    struct tm :
+        int tm_yday 
+        int tm_hour
+        int tm_min
+        int tm_sec
+
+    enum: CLOCKS_PER_SEC
+    
+    ctypedef int time_t
+    ctypedef int clock_t
+    ctypedef int suseconds_t
+    
+    struct timeval:
+        time_t      tv_sec     #  seconds */
+        suseconds_t tv_usec    #  microseconds */
+
+
+    struct timezone :
+        int tz_minuteswest;    # minutes west of Greenwich
+        int tz_dsttime;        # type of DST correction 
+
+        
+    int gettimeofday(timeval *tv, timezone *tz)
+
+
+    tm *gmtime_r(time_t *clock, tm *result)
+    time_t time(time_t *tloc)
+    clock_t clock()
+
+cdef class ProgressBar:
+    cdef off_t   maxi
+    cdef clock_t starttime
+    cdef clock_t lasttime
+    cdef clock_t tickcount
+    cdef int freq
+    cdef int cycle
+    cdef int arrow
+    cdef int lastlog
+    cdef bint ontty
+    cdef int fd
+    
+    cdef bytes head
+    cdef char *chead
+    
+    cdef object logger
+
+    cdef char  *wheel
+    cdef char  *spaces
+    cdef char*  diese 
+    
+    cdef clock_t clock(self)

python/obitools3/apps/progress.pyx

@@ -0,0 +1,138 @@
+#cython: language_level=3
+
+'''
+Created on 27 mars 2016
+
+@author: coissac
+'''
+
+import sys
+from ..utils cimport bytes2str
+
+cdef class ProgressBar:
+    cdef clock_t clock(self):
+        cdef clock_t t
+        cdef timeval tp
+        cdef clock_t s
+        
+        <void> gettimeofday(&tp,NULL)
+        s = <clock_t> (<double> tp.tv_usec * 1.e-6 * <double> CLOCKS_PER_SEC)
+        t = tp.tv_sec * CLOCKS_PER_SEC + s 
+        
+        return t
+
+    def __init__(self,
+                 off_t maxi,
+                 dict  config,
+                 str head="",
+                 double seconde=0.1):
+        self.starttime = self.clock()
+        self.lasttime  = self.starttime
+        self.tickcount = <clock_t> (seconde * CLOCKS_PER_SEC)
+        self.freq      = 1
+        self.cycle     = 0
+        self.arrow     = 0
+        self.lastlog   = 0
+        
+        self.ontty = sys.stderr.isatty()
+        
+        if (maxi<=0):
+            maxi=1
+            
+        self.maxi = maxi
+        self.head = str2bytes(head)
+        self.chead= self.head 
+        
+        
+        self.logger=config[config["__root_config__"]]["logger"]
+        self.wheel =  '|/-\\'
+        self.spaces='          ' \
+                    '          ' \
+                    '          ' \
+                    '          ' \
+                    '          '
+        self.diese ='##########' \
+                    '##########' \
+                    '##########' \
+                    '##########' \
+                    '##########'  
+                           
+    def __call__(self,object pos):
+        cdef off_t    ipos
+        cdef clock_t  elapsed
+        cdef clock_t  newtime
+        cdef clock_t  delta
+        cdef clock_t  more 
+        cdef double   percent 
+        cdef tm remain
+        cdef int days,hour,minu,sec
+        cdef off_t fraction
+        cdef int twentyth
+        
+        self.cycle+=1
+    
+        if self.cycle % self.freq == 0:
+            self.cycle=1
+            newtime  = self.clock()
+            delta         = newtime - self.lasttime
+            self.lasttime = newtime
+            elapsed       = newtime - self.starttime
+#            print(" ",delta,elapsed,elapsed/CLOCKS_PER_SEC,self.tickcount)
+            
+            if   delta < self.tickcount / 5 :
+                self.freq*=2
+            elif delta > self.tickcount * 5 and self.freq>1:
+                self.freq/=2
+                
+            
+            if callable(pos):
+                ipos=pos()
+            else:
+                ipos=pos
+                
+            if ipos==0:
+                ipos=1                
+
+            percent = <double>ipos/<double>self.maxi
+            more = <time_t>((<double>elapsed / percent * (1. - percent))/CLOCKS_PER_SEC)
+            <void>gmtime_r(&more, &remain)
+            days  = remain.tm_yday 
+            hour  = remain.tm_hour
+            minu  = remain.tm_min
+            sec   = remain.tm_sec
+                
+            if self.ontty:
+                fraction=<int>(percent * 50.)
+                self.arrow=(self.arrow+1) % 4
+                self.diese[fraction]=0
+                self.spaces[50 - fraction]=0
+            
+                if days:
+                    <void>fprintf(stderr,b'\r%s %5.1f %% |%s%c%s] remain : %d days %02d:%02d:%02d',
+                                    self.chead,
+                                    percent*100,
+                                    self.diese,self.wheel[self.arrow],self.spaces,
+                                    days,hour,minu,sec)
+                else:
+                    <void>fprintf(stderr,b'\r%s %5.1f %% |%s%c%s] remain : %02d:%02d:%02d',
+                                    self.chead,
+                                    percent*100.,
+                                    self.diese,self.wheel[self.arrow],self.spaces,
+                                    hour,minu,sec)
+                self.diese[fraction]=b'#'
+                self.spaces[50 - fraction]=b' '
+
+            twentyth = int(percent * 20)
+            if twentyth != self.lastlog:
+                
+                if self.ontty:
+                    <void>fprintf(stderr,b'\n')
+                    
+                self.logger.info('%s %5.1f %% remain : %02d:%02d:%02d' % (
+                                        bytes2str(self.head),
+                                        percent*100.,
+                                        hour,minu,sec))
+                self.lastlog=twentyth
+        else:
+            self.cycle+=1
+

python/obitools3/commands/count.py

@@ -0,0 +1,44 @@
+'''
+Created on 8 mars 2016
+
+@author: coissac
+'''
+
+from obitools3.apps.progress import ProgressBar  # @UnresolvedImport
+import time
+
+__title__="Counts sequences in a sequence set"
+
+
+default_config = {   'countmode'       : None
+                 }
+
+def addOptions(parser):
+    parser.add_argument(dest='obi:input',     metavar='obi:input', 
+                                              nargs='?', 
+                                              default=None,
+                        help='input data set' )
+
+    group=parser.add_argument_group('Obicount specific options')
+    group.add_argument('-s','--sequence',
+                             action="store_true", dest="count:sequence",
+                             default=False,
+                             help="Prints only the number of sequence records."
+                             )
+ 
+    group.add_argument('-a','--all',
+                             action="store_true", dest="count:all",
+                             default=False,
+                             help="Prints only the total count of sequence records (if a sequence has no `count` attribute, its default count is 1) (default: False)."
+                             )
+
+    
+    
+def run(config):
+    # The code of my command
+    pb = ProgressBar(1000,config,seconde=1)
+    
+    for i in range(1,1001):
+        pb(i)
+        time.sleep(0.01)
+        

python/obitools3/commands/import.pyx

@@ -0,0 +1,136 @@
+from obitools3.apps.progress cimport ProgressBar  # @UnresolvedImport
+from obitools3.files.universalopener cimport uopen
+from obitools3.parsers.fasta import fastaIterator
+from obitools3.parsers.fastq import fastqIterator
+from obitools3.obidms._obidms import OBIDMS
+
+
+import time
+
+__title__="Counts sequences in a sequence set"
+
+
+default_config = {   'destview'     : None,
+                     'skip'         : 0,
+                     'only'         : None,
+                     'skiperror'    : False,
+                     'seqinformat'  : None,
+                     'moltype'      : 'nuc',
+                     'filename'     : None
+                 }
+
+def addOptions(parser):
+    parser.add_argument(dest='import:filename',     
+                        metavar='<FILENAME>', 
+                        nargs='?', 
+                        default=None,
+                        help='sequence file name to be imported' )
+
+    group=parser.add_argument_group('obi import specific options')
+
+    group.add_argument('--default-dms','-d',
+                     action="store", dest="obi:defaultdms",
+                     metavar='<DMS NAME>',
+                     default=None,
+                     type=str,
+                     help="Name of the default DMS for reading and writing data")
+    
+    
+    group.add_argument('--destination-view','-v',
+                     action="store", dest="import:destview",
+                     metavar='<VIEW NAME>',
+                     default=None,
+                     type=str,
+                     required=True,
+                     help="Name of the default DMS for reading and writing data")
+
+    group=parser.add_argument_group('obi import specific options')
+    
+    group.add_argument('--skip',
+                     action="store", dest="import:skip",
+                     metavar='<N>',
+                     default=None,
+                     type=int,
+                     help="skip the N first sequences")
+
+    group.add_argument('--only',
+                     action="store", dest="import:only",
+                     metavar='<N>',
+                     default=None,
+                     type=int,
+                     help="treat only N sequences")
+
+    group.add_argument('--skip-on-error',
+                     action="store_true", dest="import:skiperror",
+                     default=None,
+                     help="Skip sequence entries with parse error")
+    
+    group.add_argument('--fasta',
+                     action="store_const", dest="import:seqinformat",
+                     default=None,
+                     const='fasta',
+                     help="Input file is in fasta nucleic format (including obitools fasta extentions)")
+
+    group.add_argument('--fastq',
+                     action="store_const", dest="import:seqinformat",
+                     default=None,
+                     const='fastq',
+                     help="Input file is in sanger fastq nucleic format (standard fastq)")
+
+    group.add_argument('--nuc',
+                     action="store_const", dest="import:moltype",
+                     default=None,
+                     const='nuc',
+                     help="Input file contains nucleic sequences")
+    
+    group.add_argument('--prot',
+                     action="store_const", dest="import:moltype",
+                     default=None,
+                     const='pep',
+                     help="Input file contains protein sequences")
+
+    
+    
+def run(config):
+    pb = ProgressBar(35000000,config,seconde=5)
+      
+    inputs = uopen(config['import']['filename'])
+    
+    if config['import']['seqinformat']=='fasta':
+        iseq = fastaIterator(inputs)
+        view_type="NUC_SEQS_VIEW"
+    elif config['import']['seqinformat']=='fastq':
+        iseq = fastqIterator(inputs)
+        view_type="NUC_SEQS_VIEW"
+    else:
+        raise RuntimeError('No file format specified')
+    
+    # Temporary way to handle NA values
+    #NA_list = ["nan"]
+    
+    # Create DMS
+    d = OBIDMS(config['obi']['defaultdms'])
+    
+    # Create view
+    view = d.new_view(config['import']['destview'], view_type=view_type)
+    
+    i = 0
+    for seq in iseq:
+        pb(i)
+        view[i].set_id(seq['id'])
+        view[i].set_definition(seq['definition'])
+        view[i].set_sequence(seq['sequence'])
+        for tag in seq['tags'] :
+            #print(tag, seq['tags'][tag])
+            #if seq['tags'][tag] not in NA_list :
+            view[i][tag] = seq['tags'][tag]
+        i+=1
+
+   #print(view)
+    print(view.__repr__())
+
+    view.save_and_close()
+    d.close()
+    
+    print("Done.")
+        

python/obitools3/files/linebuffer.pxd

@@ -0,0 +1,6 @@
+#cython: language_level=3
+
+
+cdef class LineBuffer:
+    cdef object fileobj
+    cdef int    size

python/obitools3/files/linebuffer.pyx

@@ -0,0 +1,23 @@
+#cython: language_level=3
+
+'''
+Created on 30 mars 2016
+
+@author: coissac
+'''
+
+cdef class LineBuffer:
+
+    def __init__(self,object fileobj,int size=100000000):
+        self.fileobj=fileobj 
+        self.size=size
+    
+    def __iter__(self):
+        cdef list buff = self.fileobj.readlines(self.size)
+        cdef str  l
+        
+        while buff:
+            for l in buff:
+                yield l
+            buff = self.fileobj.readlines(self.size)
+    

python/obitools3/files/uncompress.pxd

@@ -0,0 +1,17 @@
+#cython: language_level=3
+
+cdef class MagicKeyFile:
+    cdef object stream
+    cdef str    stream_mode
+    cdef object binary
+    cdef bytes  key 
+    cdef int    keylength 
+    cdef int    pos
+    
+    cpdef bytes read(self,int size=?)
+    cpdef int tell(self)
+
+    
+cdef class CompressedFile:
+    cdef object accessor
+    

python/obitools3/files/uncompress.pyx

@@ -0,0 +1,114 @@
+#cython: language_level=3
+
+'''
+Created on 28 mars 2016
+
+@author: coissac
+'''
+
+import zipfile
+import bz2
+import gzip
+
+import io
+
+cdef class MagicKeyFile:
+    def __init__(self,stream,length=2):
+        
+        binary=stream
+        self.stream = stream
+        self.stream_mode = None 
+        if hasattr(stream, "mode"):
+            self.stream_mode = stream.mode
+            if (not 'b' in stream.mode and 
+                hasattr(stream, "buffer") and 
+                'b' in stream.buffer.mode):
+                binary=stream.buffer
+
+        if (self.stream_mode is None and 
+            not (hasattr(stream, 'headers') and 
+                 hasattr(stream.headers, "keys") and
+                 'Content-type' in stream.headers)):
+            raise TypeError("stream does not present the good interface")
+        
+        self.binary=binary
+        self.key=binary.read(length)
+        self.keylength=length
+        self.pos=0
+        
+    cpdef bytes read(self,int size=-1):
+        cdef bytes r
+        
+        if self.pos < self.keylength:
+            if size > (self.keylength - self.pos):
+                size = size - self.keylength + self.pos
+                r = self.key[self.pos:] + self.binary.read(size)
+                self.pos=self.keylength + 1
+            elif size >=0 :
+                r = self.key[self.pos:(self.pos+size)]
+                self.pos+=size
+            else:
+                r = self.key[self.pos:] + self.binary.read(size)
+                self.pos=self.keylength + 1
+        else:
+            r = self.binary.read(size)
+            
+        return r
+    
+    cpdef int tell(self):
+        cdef int p
+        
+        if self.pos < self.keylength:
+            p = self.pos 
+        else:
+            p = self.tell()
+        
+        return p
+    
+    def __getattr__(self,name):
+        return getattr(self.binary, name)
+        
+
+    
+cdef class CompressedFile:
+    
+    
+    def __init__(self,stream):
+        cdef int            keylength
+        cdef MagicKeyFile   magic
+        cdef str            compressor
+        cdef bytes          k 
+        cdef object         c
+        
+        cdef dict compress = { 'zip' : (b'\x50\x4b\x03\x04',zipfile.ZipFile),
+                               'bz2' : (b'\x42\x5a\x68',bz2.BZ2File),
+                               'gz'  : (b'\x1f\x8b\x08',gzip.open)
+                   }
+        
+        keylength = max([len(x[0]) for x in compress.values()])
+        magic=MagicKeyFile(stream,keylength)
+        
+        self.accessor = None
+        
+        for compressor in compress:
+            k,c = compress[compressor]
+            if magic.key.startswith(k):
+                self.accessor = c(magic)
+        
+        if self.accessor is None:
+            self.accessor = magic
+        
+        if ((hasattr(stream, 'headers') and 
+             hasattr(stream.headers, "keys") and
+             'Content-type' in stream.headers and
+             stream.headers['Content-type'].startswith('text/')) or
+            'b' not in magic.stream_mode):
+            self.accessor = io.TextIOWrapper(self.accessor)
+        
+    
+    def __getattr__(self,name):
+        return getattr(self.accessor, name)
+    
+    def __iter__(self):
+        for x in self.accessor:
+            yield x 

python/obitools3/files/universalopener.pxd

@@ -0,0 +1,5 @@
+#cython: language_level=3
+
+from .uncompress cimport CompressedFile
+
+cpdef CompressedFile uopen(str name, mode=?)

python/obitools3/files/universalopener.pyx

@@ -0,0 +1,23 @@
+#cython: language_level=3
+
+'''
+Created on 25 mars 2016
+
+@author: coissac
+'''
+
+from urllib.request import urlopen
+
+
+cpdef CompressedFile uopen(str name, mode='r'):
+    cdef CompressedFile c
+    
+    try:
+        f = urlopen(name)
+    except:
+        f = open(name,mode) 
+        
+    c = CompressedFile(f)
+    
+    return c
+    

python/obitools3/obicount.py

@@ -1,31 +0,0 @@
-import sys
-import argparse
-
-
-from obitools3.obidms.obidmscolumn.capidmscolumn import OBIDMS_column
-
-
-if __name__ == '__main__': 
-    
-    parser = argparse.ArgumentParser(description='Computes the sum of a column.')
-    
-    parser.add_argument('-d', '--dms', dest='dms_name', type=str,
-                       help='Name of the OBIDMS containing the column')
-    parser.add_argument('-c', '--column', dest='column_name', type=str, default='count',
-                       help="Name of the OBIDMS column (default: 'count')")
-    parser.add_argument('-v', '--version', dest='version_number', type=int, default=-1,
-                       help='Version number of the column (default: latest version)')
-    
-    args = parser.parse_args()
-   
-    c = OBIDMS_column.open(args.dms_name, args.column_name, version_number=args.version_number)
-    
-    # check that 1 element / line and summable type?
-
-    total = 0    
-    
-    for count in c :
-        total+=count
-    
-    print("Total count = ", total)    
-

python/obitools3/obidms/_obidms.cfiles

@@ -1,5 +1,28 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
 ../../../src/obidms.h
 ../../../src/obidms.c
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
@@ -10,5 +33,19 @@
 ../../../src/obilittlebigman.c
 ../../../src/obitypes.h
 ../../../src/obitypes.c
-../../../src/private_at_functions.h
-../../../src/private_at_functions.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/utils.h
+../../../src/utils.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h

python/obitools3/obidms/_obidms.pxd

@@ -2,37 +2,93 @@
 
 from .capi.obidms       cimport OBIDMS_p
 from .capi.obidmscolumn cimport OBIDMS_column_p
-from .capi.obitypes     cimport obiversion_t, OBIType_t
+from .capi.obiview      cimport Obiview_p
+from .capi.obitypes     cimport obiversion_t, OBIType_t, index_t
+from ._obitaxo          cimport OBI_Taxonomy
+
+cdef class OBIDMS_column:
+
+    cdef  OBIDMS_column_p*   pointer
+    cdef  OBIDMS             dms
+    cdef  Obiview_p          view
+    cdef  str                data_type
+    cdef  str                dms_name
+    cdef  str                column_name
+    cdef  index_t            nb_elements_per_line
+    cdef  list               elements_names
+        
+    cpdef         update_pointer(self)
+    cpdef list    get_elements_names(self)
+    cpdef str     get_data_type(self)
+    cpdef index_t get_nb_lines_used(self)
+    cpdef str     get_creation_date(self)
+    cpdef str     get_comments(self)
+    cpdef         close(self)
+    @staticmethod
+    cdef  object  get_subclass_type(OBIDMS_column_p column_p)
+
+
+cdef class OBIDMS_column_multi_elts(OBIDMS_column):
+    
+    cpdef set_line(self, index_t line_nb, dict values)
+
+
+cdef class OBIDMS_column_line:
+
+    cdef OBIDMS_column column
+    cdef index_t index
+
+
+cdef class OBIView:
+    
+    cdef Obiview_p pointer
+    cdef str       name
+    cdef str       comments
+    cdef dict      columns
+    cdef dict      columns_pp    # TODO this dict might be unnecessary
+    cdef OBIDMS    dms
+    
+    cpdef delete_column(self, str column_name)
+    cpdef add_column(self, 
+                     str column_name, 
+                     obiversion_t version_number=*, 
+                     str type=*,
+                     index_t nb_lines=*, 
+                     index_t nb_elements_per_line=*,
+                     list elements_names=*,
+                     str indexer_name=*,
+                     str comments=*,
+                     bint create=*
+                    )
+    cpdef select_line(self, index_t line_nb)
+    cpdef select_lines(self, list line_selection)
+    cpdef save_and_close(self)
+
+
+cdef class OBIView_NUC_SEQS(OBIView):
+
+    cdef OBIDMS_column ids
+    cdef OBIDMS_column sequences
+    cdef OBIDMS_column definitions
+    
+    cpdef delete_column(self, str column_name)
+    
+
+cdef class OBIView_line :
+
+    cdef index_t index
+    cdef OBIView view
+
 
-cdef class OBIDMS_column
-            
 cdef class OBIDMS:
 
     cdef OBIDMS_p pointer
     cdef str dms_name
     
-    cpdef dict list(self)
-    
-    cpdef OBIDMS_column open_column(self,
-                                    str column_name,
-                                    bint create=*, 
-                                    bint clone=*, bint clone_data=*,
-                                    obiversion_t version_number=*, 
-                                    OBIType_t data_type=*,
-                                    size_t nb_lines=*, 
-                                    size_t nb_elements_per_line=*, 
-                                    str elements_names=*)
+    cpdef close(self)
+    cpdef OBI_Taxonomy open_taxonomy(self, str taxo_name)
+    cpdef OBIView open_view(self, str view_name)
+    cpdef OBIView new_view(self, str view_name, object view_to_clone=*, list line_selection=*, str view_type=*, str comments=*)
+    cpdef dict read_view_infos(self, str view_name)
+    cpdef dict read_views(self)
 
-
-cdef class OBIDMS_column:
-
-    cdef OBIDMS_column_p pointer
-    cdef OBIDMS dms
-    cdef str data_type   # TODO keep as OBIType_t? both?
-    cdef str dms_name
-    cdef str column_name
-
-    cpdef object get_item(self, size_t line_nb, str element_name)
-    cpdef list   get_elements_names(self)
-    cpdef str    get_data_type(self)
-    cpdef size_t get_nb_lines_used(self)

python/obitools3/obidms/_obidms.pyx

@@ -1,40 +1,636 @@
 #cython: language_level=3
 
-from pathlib import Path
-
 from obitools3.utils cimport bytes2str, str2bytes
 
-from .capi.obidms cimport obi_dms
-from .capi.obidmscolumn cimport obi_column_get_data_type_from_name, \
-                                obi_column_get_latest_version_from_name, \
-                                obi_column_get_line_count_from_name, \
-                                obi_column_get_nb_lines_used, \
-                                obi_column_get_elements_names, \
-                                obi_create_column, \
-                                obi_clone_column, \
-                                obi_open_column, \
-                                obi_close_column
-from .capi.obitypes cimport const_char_p, name_data_type
+from .capi.obidms cimport obi_dms, \
+                          obi_close_dms
+                          
+from .capi.obidmscolumn cimport obi_close_column, \
+                                OBIDMS_column_p, \
+                                OBIDMS_column_header_p
 
+from .capi.obiutils cimport obi_format_date
+                                
+from .capi.obitypes cimport const_char_p, \
+                            OBIType_t, \
+                            OBI_INT, \
+                            OBI_FLOAT, \
+                            OBI_BOOL, \
+                            OBI_CHAR, \
+                            OBI_STR, \
+                            OBI_SEQ, \
+                            name_data_type, \
+                            only_ATGC    # discuss
+                            
+from ._obidms cimport OBIDMS, \
+                      OBIDMS_column, \
+                      OBIView, \
+                      OBIView_line
 
-from ._obidms cimport OBIDMS
-from ._obidms cimport OBIDMS_column
+from ._obitaxo cimport OBI_Taxonomy
+
+from ._obiseq cimport OBI_Nuc_Seq, OBI_Nuc_Seq_Stored
 
 from ._obidmscolumn_int cimport OBIDMS_column_int, \
-                                OBIDMS_column_int_writable
-
+                                OBIDMS_column_multi_elts_int
+                                
 from ._obidmscolumn_float cimport OBIDMS_column_float, \
-                                  OBIDMS_column_float_writable
+                                  OBIDMS_column_multi_elts_float
 
 from ._obidmscolumn_bool cimport OBIDMS_column_bool, \
-                                 OBIDMS_column_bool_writable
+                                 OBIDMS_column_multi_elts_bool
 
 from ._obidmscolumn_char cimport OBIDMS_column_char, \
-                                 OBIDMS_column_char_writable
+                                 OBIDMS_column_multi_elts_char
 
-from ._obidmscolumn_idx cimport OBIDMS_column_idx, \
-                                OBIDMS_column_idx_writable
+from ._obidmscolumn_str cimport OBIDMS_column_str, \
+                                OBIDMS_column_multi_elts_str
 
+from ._obidmscolumn_seq cimport OBIDMS_column_seq, \
+                                OBIDMS_column_multi_elts_seq
+
+from .capi.obiview cimport Obiview_p, \
+                           Obiviews_infos_all_p, \
+                           Obiview_infos_p, \
+                           Column_reference_p, \
+                           obi_new_view_nuc_seqs, \
+                           obi_new_view, \
+                           obi_new_view_cloned_from_name, \
+                           obi_new_view_nuc_seqs_cloned_from_name, \
+                           obi_open_view,  \
+                           obi_read_view_infos, \
+                           obi_close_view_infos, \
+                           obi_view_delete_column, \
+                           obi_view_add_column, \
+                           obi_view_get_column, \
+                           obi_view_get_pointer_on_column_in_view, \
+                           obi_select_line, \
+                           obi_select_lines, \
+                           obi_save_and_close_view, \
+                           VIEW_TYPE_NUC_SEQS, \
+                           NUC_SEQUENCE_COLUMN, \
+                           ID_COLUMN, \
+                           DEFINITION_COLUMN
+                           
+from libc.stdlib cimport malloc
+from cpython.pycapsule cimport PyCapsule_New, PyCapsule_GetPointer
+
+
+cdef class OBIDMS_column :
+
+    # Should only be initialized through a subclass
+    def __init__(self, OBIView view, str column_name):
+        
+        cdef OBIDMS_column_p  column_p
+        cdef OBIDMS_column_p* column_pp
+        
+        column_pp = <OBIDMS_column_p*> PyCapsule_GetPointer(((view.columns_pp)[column_name]), NULL) # or use C function
+        column_p = column_pp[0]     # TODO ugly cython dereferencing but can't find better
+        
+        # Fill structure
+        self.pointer = column_pp
+        self.dms = view.dms
+        self.view = view.pointer    # TODO pointer or instance?
+        self.data_type = bytes2str(name_data_type((column_p.header).returned_data_type))
+        self.column_name = bytes2str((column_p.header).name)
+        self.nb_elements_per_line = (column_p.header).nb_elements_per_line
+        self.elements_names = (bytes2str((column_p.header).elements_names)).split(';')
+
+    def __setitem__(self, index_t line_nb, object value):
+        self.set_line(line_nb, value)
+
+    def __getitem__(self, index_t line_nb):
+        return self.get_line(line_nb)
+
+    def __len__(self):
+        return (self.pointer)[0].header.lines_used
+    
+    def __sizeof__(self):
+        return ((self.pointer)[0].header.header_size + (self.pointer)[0].header.data_size)
+    
+    def __iter__(self):
+    
+        # Declarations
+        cdef index_t lines_used
+        cdef index_t line_nb
+        
+        # Yield each line
+        lines_used = (self.pointer)[0].header.lines_used
+        for line_nb in range(lines_used):
+            yield self.get_line(line_nb)
+    
+    cpdef update_pointer(self):
+        self.pointer = <OBIDMS_column_p*> obi_view_get_pointer_on_column_in_view(self.view, str2bytes(self.column_name))
+
+    cpdef list get_elements_names(self):
+        return self.elements_names
+         
+    cpdef str get_data_type(self):
+        return self.data_type
+    
+    cpdef index_t get_nb_lines_used(self):
+        return (self.pointer)[0].header.lines_used
+        
+    cpdef str get_creation_date(self):
+        return bytes2str(obi_format_date((self.pointer)[0].header.creation_date))
+    
+    cpdef str get_comments(self):
+        return bytes2str((self.pointer)[0].header.comments)
+    
+    def __repr__(self) :
+        cdef str to_print
+        to_print = ''
+        for line in self :
+            to_print = to_print + str(line) + "\n"
+        return to_print
+        
+    cpdef close(self):
+        if obi_close_column((self.pointer)[0]) < 0 :
+            raise Exception("Problem closing a column")
+
+
+    @staticmethod
+    cdef object get_subclass_type(OBIDMS_column_p column_p) :
+        
+        cdef object subclass
+        cdef OBIDMS_column_header_p header
+        cdef OBIType_t col_type
+        cdef bint col_writable
+        cdef bint col_one_element_per_line
+
+        header = column_p.header        
+        col_type = header.returned_data_type
+        col_writable = column_p.writable
+        col_one_element_per_line = ((header.nb_elements_per_line) == 1)
+        
+        if col_type == OBI_INT :
+            if col_one_element_per_line :
+                subclass = OBIDMS_column_int
+            else :
+                subclass = OBIDMS_column_multi_elts_int
+        elif col_type == OBI_FLOAT :
+            if col_one_element_per_line :
+                subclass = OBIDMS_column_float
+            else :
+                subclass = OBIDMS_column_multi_elts_float
+        elif col_type == OBI_BOOL :
+            if col_one_element_per_line :
+                subclass = OBIDMS_column_bool
+            else :
+                subclass = OBIDMS_column_multi_elts_bool
+        elif col_type == OBI_CHAR :
+            if col_one_element_per_line :
+                subclass = OBIDMS_column_char
+            else :
+                subclass = OBIDMS_column_multi_elts_char
+        elif col_type == OBI_STR :
+            if col_one_element_per_line :
+                subclass = OBIDMS_column_str
+            else :
+                subclass = OBIDMS_column_multi_elts_str
+        elif col_type == OBI_SEQ :
+            if col_one_element_per_line :
+                subclass = OBIDMS_column_seq
+            else :
+                subclass = OBIDMS_column_multi_elts_seq
+        else :
+            raise Exception("Problem with the data type")
+
+        return subclass
+
+    
+######################################################################################################
+
+cdef class OBIDMS_column_multi_elts(OBIDMS_column) :
+    
+    def __getitem__(self, index_t line_nb):
+        return OBIDMS_column_line(self, line_nb)
+
+    cpdef set_line(self, index_t line_nb, dict values):
+        for element_name in values :
+            self.set_item(line_nb, element_name, values[element_name])
+
+######################################################################################################
+
+cdef class OBIDMS_column_line :
+
+    def __init__(self, OBIDMS_column column, index_t line_nb) :
+        self.index = line_nb
+        self.column = column
+        
+    def __getitem__(self, str element_name) :
+        return self.column.get_item(self.index, element_name)
+     
+    def __setitem__(self, str element_name, object value):
+        self.column.set_item(self.index, element_name, value)
+
+    def __contains__(self, str element_name):
+        return (element_name in self.column.elements_names)
+
+    def __repr__(self) :
+        return str(self.column.get_line(self.index))
+
+##########################################
+
+
+cdef class OBIView :
+    
+    def __init__(self, OBIDMS dms, str view_name, bint new=False, object view_to_clone=None, list line_selection=None, str comments=""): 
+        
+        cdef Obiview_p view = NULL
+        cdef int i
+        cdef list col_list
+        cdef str col_name
+        cdef OBIDMS_column column
+        cdef OBIDMS_column_p column_p
+        cdef OBIDMS_column_p* column_pp
+        cdef OBIDMS_column_header_p header
+        cdef index_t* line_selection_p
+        cdef object col_capsule
+        
+        self.dms = dms
+        
+        if line_selection is not None :
+            line_selection_p = <index_t*> malloc((len(line_selection) + 1) * sizeof(index_t))
+            for i in range(len(line_selection)) :
+                line_selection_p[i] = line_selection[i]     # TODO type problem?
+            line_selection_p[len(line_selection)] = -1
+        else :
+            line_selection_p = NULL
+       
+        if new :
+            if view_to_clone is not None :
+                if type(view_to_clone) == str :
+                    view = obi_new_view_cloned_from_name(dms.pointer, str2bytes(view_name), str2bytes(view_to_clone), line_selection_p, str2bytes(comments))
+                else :
+                    view = obi_new_view(dms.pointer, str2bytes(view_name), (<OBIView> view_to_clone).pointer, line_selection_p, str2bytes(comments))
+            elif view_to_clone is None :
+                view = obi_new_view(dms.pointer, str2bytes(view_name), NULL, line_selection_p, str2bytes(comments))
+        elif not new :
+            if view_name is not None :
+                view = obi_open_view(dms.pointer, str2bytes(view_name))
+            elif view_name is None :
+                view = obi_open_view(dms.pointer, NULL)
+        
+        if view == NULL :
+            raise Exception("Error creating/opening view")
+
+        self.pointer = view
+        self.name = bytes2str(view.name)
+        
+        # go through columns to build list and open python object (TODO make separate function?)
+        self.columns = {}
+        self.columns_pp = {}
+                
+        i = 0
+        while i < view.column_count :
+            column_pp = <OBIDMS_column_p*> ((view.columns)+i)
+            column_p = <OBIDMS_column_p> (view.columns)[i]
+            header = (column_p).header
+            
+            col_name = bytes2str(header.name)
+            col_capsule = PyCapsule_New(column_pp, NULL, NULL)   # TODO discuss
+            (self.columns_pp)[col_name] = col_capsule
+            
+            subclass = OBIDMS_column.get_subclass_type(column_p)
+            self.columns[col_name] = subclass(self, col_name)
+            
+            i+=1
+
+
+    def __repr__(self) :
+        
+        cdef str s
+        cdef OBIDMS_column column
+        cdef OBIDMS_column_p column_p
+        
+        s = self.name
+        s = s + ", " + self.comments + ", " + str(self.pointer.line_count) + " lines" 
+        for column_name in self.columns :   # TODO make function in OBIDMS_column class
+            column = self.columns[column_name]
+            column_p = (column.pointer)[0]
+            s = s + "\n" + column_name + ", version " + str(column_p.header.version) + ", data type: " + column.data_type
+        return s
+        
+    
+    cpdef delete_column(self, str column_name) :
+        
+        cdef int i
+        cdef Obiview_p view
+        cdef OBIDMS_column column
+        cdef OBIDMS_column_p column_p
+        cdef OBIDMS_column_p* column_pp
+        cdef OBIDMS_column_header_p header
+        cdef str column_n
+ 
+        view = self.pointer
+ 
+        if obi_view_delete_column(view, str2bytes(column_name)) < 0 :
+            raise Exception("Problem deleting a column from a view")
+        
+        # Update the dictionaries of column pointers and column objects, and update pointers in column objects (make function?):
+        (self.columns).pop(column_name)
+        (self.columns_pp).pop(column_name)
+        
+        i = 0
+        while i < view.column_count :
+            column_pp = <OBIDMS_column_p*> ((view.columns)+i)
+            column_p = <OBIDMS_column_p> (view.columns)[i]
+            header = (column_p).header
+            col_name = bytes2str(header.name)
+            col_capsule = PyCapsule_New(column_pp, NULL, NULL)
+            (self.columns_pp)[col_name] = col_capsule
+            i+=1
+
+        for column_n in self.columns :
+            (self.columns[column_n]).update_pointer()
+
+    
+    cpdef add_column(self, 
+                     str column_name, 
+                     obiversion_t version_number=-1, 
+                     str type='',
+                     index_t nb_lines=0, 
+                     index_t nb_elements_per_line=1,    # TODO 1?
+                     list elements_names=None,
+                     str indexer_name="",
+                     str comments="",
+                     bint create=True   # TODO
+                     ) :
+        
+        cdef bytes column_name_b
+        cdef bytes elements_names_b
+        cdef object subclass
+        cdef OBIDMS_column_p* column_pp
+        cdef OBIDMS_column_p column_p
+        
+        column_name_b = str2bytes(column_name)
+        
+        if nb_elements_per_line > 1 :
+            elements_names_b = str2bytes(';'.join(elements_names))
+        elif nb_elements_per_line == 1 :
+            elements_names_b = column_name_b
+    
+        if type :    
+            if type == 'OBI_INT' :
+                data_type = OBI_INT
+            elif type == 'OBI_FLOAT' :
+                data_type = OBI_FLOAT
+            elif type == 'OBI_BOOL' :
+                data_type = OBI_BOOL
+            elif type == 'OBI_CHAR' :
+                data_type = OBI_CHAR
+            elif type == 'OBI_STR' :
+                data_type = OBI_STR
+            elif type == 'OBI_SEQ' :
+                data_type = OBI_SEQ
+            else :
+                raise Exception("Invalid provided data type")
+        
+        if (obi_view_add_column(self.pointer, column_name_b, version_number,     # should return pointer on column?
+                                data_type, nb_lines, nb_elements_per_line, 
+                                elements_names_b, str2bytes(indexer_name), 
+                                str2bytes(comments), create) < 0) :
+            raise Exception("Problem adding a column in a view")
+        
+        # Store the column pointer
+        column_pp = obi_view_get_pointer_on_column_in_view(self.pointer, column_name_b)
+        if column_pp == NULL :
+            raise Exception("Problem getting a column in a view")
+        
+        col_capsule = PyCapsule_New(column_pp, NULL, NULL)   # TODO
+        (self.columns_pp)[column_name] = col_capsule
+        
+        # Open and store the subclass
+        column_p = column_pp[0]     # TODO ugly cython dereferencing
+        subclass = OBIDMS_column.get_subclass_type(column_p)
+        (self.columns)[column_name] = subclass(self, column_name)
+                
+    
+    cpdef save_and_close(self) :
+        if (obi_save_and_close_view(self.pointer) < 0) :
+            raise Exception("Problem closing a view")
+    
+
+    def __iter__(self):
+        # iter on each line of all columns
+        
+        # Declarations
+        cdef index_t lines_used
+        cdef index_t line_nb
+        cdef OBIView_line line  # TODO for NUC SEQS View
+        
+        # Yield each line TODO line class
+        lines_used = (self.pointer).line_count
+        
+        for line_nb in range(lines_used) :
+            line = self[line_nb]
+            yield line
+
+
+    def __getitem__(self, object item) :
+        if type(item) == str :
+            return (self.columns)[item]
+        elif type(item) == int :     # TODO int?
+            return OBIView_line(self, item)
+
+
+    cpdef select_line(self, index_t line_nb) :
+        if obi_select_line(self.pointer, line_nb) < 0 :
+            raise Exception("Problem selecting a line")
+
+    
+    cpdef select_lines(self, list line_selection) :
+        cdef index_t* line_selection_p
+        line_selection_p = <index_t*> malloc((len(line_selection) + 1) * sizeof(index_t))
+        for i in range(len(line_selection)) :
+            line_selection_p[i] = line_selection[i]     # TODO type problem?
+        line_selection_p[len(line_selection)] = -1
+        if obi_select_lines(self.pointer, line_selection_p) < 0 :
+            raise Exception("Problem selecting a list of lines")
+    
+    
+    def __contains__(self, str column_name):
+        return (column_name in self.columns)
+    
+    
+    def __str__(self) :
+        cdef OBIView_line line
+        cdef str to_print
+        to_print = ""
+        for line in self.__iter__() :
+            to_print = to_print + str(line) + "\n"
+        return to_print
+
+#############################################
+
+cdef class OBIView_NUC_SEQS(OBIView):
+    
+    def __init__(self, OBIDMS dms, str view_name, bint new=False, object view_to_clone=None, list line_selection=None, str comments=""): 
+        
+        cdef Obiview_p view = NULL
+        cdef int i
+        cdef list col_list
+        cdef str col_name
+        cdef OBIDMS_column column
+        cdef OBIDMS_column_p column_p
+        cdef OBIDMS_column_p* column_pp
+        cdef OBIDMS_column_header_p header
+        cdef index_t* line_selection_p
+                
+        self.dms = dms
+        
+        if line_selection is not None :
+            line_selection_p = <index_t*> malloc((len(line_selection) + 1) * sizeof(index_t))
+            for i in range(len(line_selection)) :
+                line_selection_p[i] = line_selection[i]     # TODO type problem?
+            line_selection_p[len(line_selection)] = -1
+        else :
+            line_selection_p = NULL
+       
+        if new :
+            if view_to_clone is not None :
+                if type(view_to_clone) == str :
+                    view = obi_new_view_nuc_seqs_cloned_from_name(dms.pointer, str2bytes(view_name), str2bytes(view_to_clone), line_selection_p, str2bytes(comments))
+                else :
+                    view = obi_new_view_nuc_seqs(dms.pointer, str2bytes(view_name), (<OBIView> view_to_clone).pointer, line_selection_p, str2bytes(comments))
+            elif view_to_clone is None :
+                view = obi_new_view_nuc_seqs(dms.pointer, str2bytes(view_name), NULL, line_selection_p, str2bytes(comments))
+        elif not new :
+            if view_name is not None :
+                view = obi_open_view(dms.pointer, str2bytes(view_name))
+            elif view_name is None :
+                view = obi_open_view(dms.pointer, NULL)
+        
+        if view == NULL :
+            raise Exception("Error creating/opening view")
+
+        self.pointer = view
+        self.name = bytes2str(view.name)
+        self.comments = bytes2str(view.comments)
+        
+        # go through columns to build list and open python object (TODO make separate function?)
+        self.columns = {}
+        self.columns_pp = {}
+                
+        i = 0
+        while i < view.column_count :
+            column_pp = <OBIDMS_column_p*> ((view.columns)+i)
+            column_p = <OBIDMS_column_p> (view.columns)[i]
+            header = (column_p).header
+            
+            col_name = bytes2str(header.name)
+            col_capsule = PyCapsule_New(column_pp, NULL, NULL)   # TODO discuss
+            (self.columns_pp)[col_name] = col_capsule
+            
+            subclass = OBIDMS_column.get_subclass_type(column_p)
+            self.columns[col_name] = subclass(self, col_name)
+            
+            i+=1
+        
+        self.ids = self.columns[bytes2str(ID_COLUMN)]
+        self.sequences = self.columns[bytes2str(NUC_SEQUENCE_COLUMN)]
+        self.definitions = self.columns[bytes2str(DEFINITION_COLUMN)]
+
+
+    cpdef delete_column(self, str column_name) :
+        
+        cdef int i
+        cdef Obiview_p view
+        cdef OBIDMS_column column
+        cdef OBIDMS_column_p column_p
+        cdef OBIDMS_column_p* column_pp
+        cdef OBIDMS_column_header_p header
+        cdef str column_n
+ 
+        if ((column_name == bytes2str(ID_COLUMN)) or (column_name == bytes2str(NUC_SEQUENCE_COLUMN)) or (column_name == bytes2str(DEFINITION_COLUMN))) :
+            raise Exception("Can't delete an obligatory column from a NUC_SEQS view")
+        
+        view = self.pointer
+ 
+        if obi_view_delete_column(view, str2bytes(column_name)) < 0 :
+            raise Exception("Problem deleting a column from a view")
+        
+        # Update the dictionaries of column pointers and column objects, and update pointers in column objects (make function?):
+        (self.columns).pop(column_name)
+        (self.columns_pp).pop(column_name)
+        
+        i = 0
+        while i < view.column_count :
+            column_pp = <OBIDMS_column_p*> ((view.columns)+i)
+            column_p = <OBIDMS_column_p> (view.columns)[i]
+            header = (column_p).header
+            col_name = bytes2str(header.name)
+            col_capsule = PyCapsule_New(column_pp, NULL, NULL)
+            (self.columns_pp)[col_name] = col_capsule
+            i+=1
+
+        for column_n in self.columns :
+            (self.columns[column_n]).update_pointer()
+
+            
+    def __getitem__(self, object item) :
+        if type(item) == str :
+            return (self.columns)[item]
+        elif type(item) == int :     # TODO int?
+            return OBI_Nuc_Seq_Stored(self, item)
+    
+    
+    def __setitem__(self, index_t line_idx, OBI_Nuc_Seq sequence_obj) :
+        for key in sequence_obj :
+            self[line_idx][key] = sequence_obj[key]
+            
+
+#############################################
+
+cdef class OBIView_line :
+
+    def __init__(self, OBIView view, index_t line_nb) :
+        self.index = line_nb
+        self.view = view
+    
+    def __getitem__(self, str column_name) :
+        return ((self.view).columns)[column_name][self.index]
+    
+    def __setitem__(self, str column_name, object value):
+        # TODO detect multiple elements (dict type)? put somewhere else? but more risky (in get)
+        cdef type value_type
+        cdef str value_obitype
+        if column_name not in self.view :
+            if value == None :
+                raise Exception("Trying to create a column from a None value (can't guess type)")
+            value_type = type(value)
+            if value_type == int :
+                value_obitype = 'OBI_INT'
+            elif value_type == float :
+                value_obitype = 'OBI_FLOAT'
+            elif value_type == bool :
+                value_obitype = 'OBI_BOOL'            
+            elif value_type == str :
+                if only_ATGC(str2bytes(value)) :    # TODO
+                    value_obitype = 'OBI_SEQ'
+                elif len(value) == 1 :
+                    value_obitype = 'OBI_CHAR'
+                elif (len(value) > 1) :
+                    value_obitype = 'OBI_STR'
+            else :
+                raise Exception("Could not guess the type of a value to create a new column")
+            self.view.add_column(column_name, type=value_obitype)
+
+        (((self.view).columns)[column_name]).set_line(self.index, value)
+
+    def __contains__(self, str column_name):
+        return (column_name in self.view)
+        
+    def __repr__(self):
+        cdef dict line
+        cdef str column_name
+        line = {}
+        for column_name in self.view.columns :
+            line[column_name] = self[column_name]
+        return str(line)
+
+##########################################
 
 cdef class OBIDMS :
     
@@ -49,235 +645,97 @@ cdef class OBIDMS :
         # Fill structure and create or open the DMS
         self.dms_name = dms_name
         self.pointer = obi_dms(<const_char_p> dms_name_b)
+        if self.pointer == NULL :
+            raise Exception("Failed opening or creating an OBIDMS")
 
 
-#    def __del__(self) :                        # TODO problem with closing dir breaking everything
-#        obi_close_dms(self.pointer)
+    cpdef close(self) :
+        if (obi_close_dms(self.pointer)) < 0 :
+            raise Exception("Problem closing an OBIDMS")
 
-
-    cpdef dict list(self):
     
-        # Declarations
-        cdef object p
-        cdef dict dms = {}
+    cpdef OBI_Taxonomy open_taxonomy(self, str taxo_name) :
+        return OBI_Taxonomy(self, taxo_name)
+    
+
+    cpdef OBIView open_view(self, str view_name) :
+        
+        cdef object view_class
+        cdef dict view_infos
+        
+        view_infos = self.read_view_infos(view_name)
+        
+        if view_infos["view_type"] == bytes2str(VIEW_TYPE_NUC_SEQS) :
+            view_class = OBIView_NUC_SEQS
+        else :
+            view_class = OBIView
+        
+        return view_class(self, view_name)
+    
+
+    cpdef OBIView new_view(self, str view_name, object view_to_clone=None, list line_selection=None, str view_type=None, str comments="") :
+        
+        cdef object view_class
+        
+        if view_type is not None :
+            if view_type == bytes2str(VIEW_TYPE_NUC_SEQS) :
+                view_class = OBIView_NUC_SEQS
+        else :
+            view_class = OBIView
+        
+        return view_class(self, view_name, new=True, view_to_clone=view_to_clone, line_selection=line_selection, comments=comments)
+
+
+    cpdef dict read_view_infos(self, str view_name) :
+        all_views = self.read_views()
+        return all_views[view_name]
+
+
+    cpdef dict read_views(self) :    # TODO function that prints the dic nicely and function that prints 1 view. Add column type in col ref
+        
+        cdef Obiviews_infos_all_p all_views_p
+        cdef Obiview_infos_p view_p
+        cdef Column_reference_p column_refs
+        cdef int nb_views
+        cdef int i, j
+        cdef str view_name
         cdef str column_name
-        cdef bytes column_name_b
-        cdef str data_type
-        cdef obiversion_t latest_version
-        cdef size_t line_count
+        cdef dict views
+        cdef bytes name_b
         
-        p = Path(self.dms_name+'.obidms')
-
-        print("{:<25} {:<25} {:<25} {:<25}".format('-Column name-','-Data type-','-Latest version number-', '-Line count of latest version-'))
-        for entry in p.iterdir():
-            if entry.suffix == ".obicol":
-                column_name = entry.stem
-                column_name_b = str2bytes(column_name)
-                dms[column_name] = {}
-                data_type = bytes2str(name_data_type(obi_column_get_data_type_from_name(self.pointer, column_name_b)))
-                latest_version = obi_column_get_latest_version_from_name(self.pointer, column_name_b)
-                line_count = obi_column_get_line_count_from_name(self.pointer, column_name_b)
-                dms[column_name]['data_type'] = data_type
-                dms[column_name]['latest_version'] = latest_version
-                dms[column_name]['line_count'] = line_count
-                print("{:<25} {:<25} {:<25} {:<25}".format(column_name, data_type, latest_version, line_count))
-
-        return dms
-     
-    cpdef OBIDMS_column open_column(self,
-                                    str column_name,
-                                    bint create=False, 
-                                    bint clone=False, bint clone_data=True,
-                                    obiversion_t version_number=-1, 
-                                    OBIType_t data_type= <OBIType_t> 0,
-                                    size_t nb_lines=0, 
-                                    size_t nb_elements_per_line=1, 
-                                    str elements_names=None):
-        
-        # Declarations
-        cdef OBIDMS_column column
-        cdef bytes column_name_b
-        
-        # Format the character string to send to C function
-        column_name_b = str2bytes(column_name)
-        
-        # Get the data type if not provided
-        if not data_type :
-            if create :
-                raise Exception("A data type must be specified")
+        views = {}
+        all_views_p = obi_read_view_infos(self.pointer)
+        if all_views_p == NULL :
+            raise Exception("No views to read")
+        nb_views = <int> (all_views_p.header).view_count
+        for i in range(nb_views) :
+            view_p = (<Obiview_infos_p> (all_views_p.view_infos)) + i
+            view_name = bytes2str(view_p.name)
+            views[view_name] = {}
+            views[view_name]["comments"] = bytes2str(view_p.comments)
+            views[view_name]["view_type"] = bytes2str(view_p.view_type)
+            views[view_name]["column_count"] = <int> view_p.column_count
+            views[view_name]["line_count"] = <int> view_p.line_count
+            views[view_name]["view_number"] = <int> view_p.view_number
+            views[view_name]["created_from"] = bytes2str(view_p.created_from)
+            views[view_name]["creation_date"] = bytes2str(obi_format_date(view_p.creation_date))     # TODO move this function in utils or somethings
+            if (view_p.all_lines) :
+                views[view_name]["line_selection"] = None
             else :
-                data_type = obi_column_get_data_type_from_name(self.pointer, column_name_b)
-
-        # Open the column with the right subclass depending on the data type and the mode (read-only or writable)
-        if data_type == 1 :
-            if (create or clone) :
-                column = OBIDMS_column_int_writable(self, column_name, 
-                                                    create, clone, clone_data, 
-                                                    version_number, data_type, 
-                                                    nb_lines, nb_elements_per_line, 
-                                                    elements_names)
-            else :
-                 column = OBIDMS_column_int(self, column_name, 
-                                                 create, clone, clone_data, 
-                                                 version_number, data_type, 
-                                                 nb_lines, nb_elements_per_line, 
-                                                 elements_names)
-               
-        elif data_type == 2 :
-            if (create or clone) :
-                column = OBIDMS_column_float_writable(self, column_name, 
-                                                      create, clone, clone_data, 
-                                                      version_number, data_type, 
-                                                      nb_lines, nb_elements_per_line, 
-                                                      elements_names)
-            else :
-                 column = OBIDMS_column_float(self, column_name, 
-                                                   create, clone, clone_data, 
-                                                   version_number, data_type, 
-                                                   nb_lines, nb_elements_per_line, 
-                                                   elements_names)
-
-        elif data_type == 3 :
-            if (create or clone) :
-                column = OBIDMS_column_bool_writable(self, column_name, 
-                                                     create, clone, clone_data, 
-                                                     version_number, data_type, 
-                                                     nb_lines, nb_elements_per_line, 
-                                                     elements_names)
-            else :
-                 column = OBIDMS_column_bool(self, column_name, 
-                                                  create, clone, clone_data, 
-                                                  version_number, data_type, 
-                                                  nb_lines, nb_elements_per_line, 
-                                                  elements_names)
-
-        elif data_type == 4 :
-            if (create or clone) :
-                column = OBIDMS_column_char_writable(self, column_name, 
-                                                     create, clone, clone_data, 
-                                                     version_number, data_type, 
-                                                     nb_lines, nb_elements_per_line, 
-                                                     elements_names)
-            else :
-                 column = OBIDMS_column_char(self, column_name, 
-                                                  create, clone, clone_data, 
-                                                  version_number, data_type, 
-                                                  nb_lines, nb_elements_per_line, 
-                                                  elements_names)
-    
-#         elif data_type == 5 :
-#             if (create or clone) :
-#                 column = OBIDMS_column_idx_writable(self, column_name, 
-#                                                     create, clone, clone_data, 
-#                                                     version_number, data_type, 
-#                                                     nb_lines, nb_elements_per_line, 
-#                                                     elements_names)
-#             else :
-#                  column = OBIDMS_column_idx(self, column_name, 
-#                                             create, clone, clone_data, 
-#                                             version_number, data_type, 
-#                                             nb_lines, nb_elements_per_line, 
-#                                             elements_names)
-   
-        else :
-            raise Exception("Problem with the data type")
-            
-        return column
-
-
-
-cdef class OBIDMS_column :
-
-    # Should only be initialized through a subclass
-    def __init__(self, 
-                 OBIDMS dms, 
-                 str column_name,
-                 bint create, 
-                 bint clone, bint clone_data,
-                 obiversion_t version_number, 
-                 OBIType_t type,
-                 size_t nb_lines, 
-                 size_t nb_elements_per_line, 
-                 str elements_names):
+                views[view_name]["line_selection"] = {}
+                views[view_name]["line_selection"]["column_name"] = bytes2str((view_p.line_selection).column_name)
+                views[view_name]["line_selection"]["version"] = <int> (view_p.line_selection).version
+            views[view_name]["column_references"] = {}
+            column_refs = view_p.column_references
+            for j in range(views[view_name]["column_count"]) :
+                column_name = bytes2str((column_refs[j]).column_name)
+                views[view_name]["column_references"][column_name] = {}
+                views[view_name]["column_references"][column_name]["version"] = column_refs[j].version
         
-        # Declarations
-        cdef bytes column_name_b
-        cdef bytes dms_name_b
-        cdef bytes elements_names_b
+        obi_close_view_infos(all_views_p);
         
-        # Fill structure
-        self.dms = dms
-        self.data_type = bytes2str(name_data_type(type))
-        self.column_name = column_name
-        
-        # Format the character strings to send them to C functions
-        column_name_b = str2bytes(column_name)
-        dms_name_b = str2bytes(self.dms.dms_name)
-        
-        # Create, clone or open column
-        if create :
-            if elements_names == None :  
-                elements_names_b = column_name_b
-            else :
-                elements_names_b = str2bytes(elements_names)
-            self.pointer = obi_create_column(self.dms.pointer, column_name_b, type, nb_lines, nb_elements_per_line, elements_names_b)
-        else :
-            if clone :
-                self.pointer = obi_clone_column(self.dms.pointer, column_name_b, version_number, clone_data)
-            else :
-                self.pointer = obi_open_column(self.dms.pointer, column_name_b, version_number)
+        return views
     
     
-    def __iter__(self):
     
-        # Declarations
-        cdef list elements_names
-        cdef str element_name
-        cdef bint multiple_elements
-        cdef object line               # TODO
-        cdef size_t lines_used
-        cdef size_t line_nb
-        
-        # Check if there are multiple elements per line anf if yes, get their names
-        elements_names = self.get_elements_names()
-        if len(elements_names) > 1 :
-            multiple_elements = True
-        else :
-            element_name = elements_names[0]
-        
-        # Yield each line
-        lines_used = obi_column_get_nb_lines_used(self.pointer)
-        for line_nb in xrange(lines_used):
-            if multiple_elements :
-                line = []
-                for element_name in elements_names :
-                    line.append(self.get_item(line_nb, element_name))
-            else :
-                line = self.get_item(line_nb, element_name)
-            yield line
-
-
-    def __setitem__(self, size_t line_nb, object value):
-        self.set_item(line_nb, "", value)
-
-        
-    def __getitem__(self, size_t line_nb):
-        return self.get_item(line_nb, "")
-
-    cpdef object get_item(self, size_t line_nb, str element_name):
-        raise NotImplementedError
-    
-    cpdef list get_elements_names(self):
-        cdef bytes elements_names
-        elements_names = obi_column_get_elements_names(self.pointer)
-        return (bytes2str(elements_names)).split(';')
-
-             
-    cpdef str get_data_type(self):
-        return self.data_type
-
-    
-    cpdef size_t get_nb_lines_used(self):
-        return obi_column_get_nb_lines_used(self.pointer)
-        
-   
+    

python/obitools3/obidms/_obidmscolumn_bool.cfiles

@@ -1,16 +1,51 @@
-../../../src/obidmscolumn_bool.c
-../../../src/obidmscolumn_bool.h
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
 ../../../src/obidmscolumndir.c
-../../../src/obidms.h
-../../../src/obidms.c
 ../../../src/obierrno.h
 ../../../src/obierrno.c
 ../../../src/obilittlebigman.h
 ../../../src/obilittlebigman.c
 ../../../src/obitypes.h
 ../../../src/obitypes.c
-../../../src/private_at_functions.h
-../../../src/private_at_functions.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/utils.h
+../../../src/utils.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h

python/obitools3/obidms/_obidmscolumn_bool.pxd

@@ -1,17 +1,14 @@
 #cython: language_level=3
 
-from .capi.obitypes cimport obibool_t
-from .capi.obidmscolumn cimport OBIDMS_column_p
-                                                
-from ._obidms cimport OBIDMS_column
+from .capi.obitypes cimport index_t
+from ._obidms       cimport OBIDMS_column, OBIDMS_column_multi_elts
 
 
 cdef class OBIDMS_column_bool(OBIDMS_column):
-    cpdef object get_item(self, size_t line_nb, str element_name)
-    cpdef set_item(self, size_t line_nb, str element_name, obibool_t value)
-    cpdef close(self)
-
-cdef class OBIDMS_column_bool_writable(OBIDMS_column_bool):
-    cpdef set_item(self, size_t line_nb, str element_name, obibool_t value)
-    cpdef close(self)
+    cpdef object get_line(self, index_t line_nb)
+    cpdef set_line(self, index_t line_nb, object value)
 
+cdef class OBIDMS_column_multi_elts_bool(OBIDMS_column_multi_elts):
+    cpdef object get_item(self, index_t line_nb, str element_name)
+    cpdef object get_line(self, index_t line_nb)
+    cpdef set_item(self, index_t line_nb, str element_name, object value)

python/obitools3/obidms/_obidmscolumn_bool.pyx

@@ -1,11 +1,11 @@
 #cython: language_level=3
  
-from .capi.obidmscolumn cimport obi_close_column,\
-                                obi_truncate_and_close_column, \
-                                obi_column_get_obibool_with_elt_name, \
-                                obi_column_set_obibool_with_elt_name
+from .capi.obiview      cimport obi_column_get_obibool_with_elt_name_in_view, \
+                                obi_column_get_obibool_with_elt_idx_in_view, \
+                                obi_column_set_obibool_with_elt_name_in_view, \
+                                obi_column_set_obibool_with_elt_idx_in_view
 from .capi.obierrno     cimport obi_errno
-from .capi.obitypes     cimport OBIBool_NA
+from .capi.obitypes     cimport OBIBool_NA, obibool_t
 
 from obitools3.utils cimport str2bytes
 
@@ -13,11 +13,32 @@ from cpython.bool cimport PyBool_FromLong
 
 
 cdef class OBIDMS_column_bool(OBIDMS_column):
- 
-    cpdef object get_item(self, size_t line_nb, str element_name):
+
+    cpdef object get_line(self, index_t line_nb):
         cdef obibool_t value
         cdef object result
-        value = obi_column_get_obibool_with_elt_name(self.pointer, line_nb, str2bytes(element_name))
+        value = obi_column_get_obibool_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0)
+        if obi_errno > 0 :
+            raise IndexError(line_nb)
+        if value == OBIBool_NA :
+            result = None
+        else :
+            result = PyBool_FromLong(value)
+        return result
+    
+    cpdef set_line(self, index_t line_nb, object value):
+        if value is None :
+            value = OBIBool_NA
+        if obi_column_set_obibool_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0, <obibool_t> value) < 0:
+            raise Exception("Problem setting a value in a column")
+
+
+cdef class OBIDMS_column_multi_elts_bool(OBIDMS_column_multi_elts):
+
+    cpdef object get_item(self, index_t line_nb, str element_name):
+        cdef obibool_t value
+        cdef object result
+        value = obi_column_get_obibool_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name))
         if obi_errno > 0 :
             raise IndexError(line_nb, element_name)
         if value == OBIBool_NA :
@@ -25,22 +46,32 @@ cdef class OBIDMS_column_bool(OBIDMS_column):
         else :
             result = PyBool_FromLong(value)
         return result
-     
-    cpdef set_item(self, size_t line_nb, str element_name, obibool_t value):
-        raise Exception("Column is read-only")
-     
-    cpdef close(self):
-        if obi_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
+    
+    cpdef object get_line(self, index_t line_nb) :
+        cdef obibool_t value
+        cdef object value_in_result
+        cdef dict result
+        cdef index_t i
+        cdef bint all_NA
+        result = {}
+        all_NA = True
+        for i in range(self.nb_elements_per_line) :
+            value = obi_column_get_obibool_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, i)
+            if obi_errno > 0 :
+                raise IndexError(line_nb)
+            if value == OBIBool_NA :
+                value_in_result = None
+            else :
+                value_in_result = PyBool_FromLong(value)   
+            result[self.elements_names[i]] = value_in_result
+            if all_NA and (value_in_result is not None) :
+                all_NA = False
+        if all_NA :
+            result = None
+        return result
 
-
-cdef class OBIDMS_column_bool_writable(OBIDMS_column_bool):     
-   
-    cpdef set_item(self, size_t line_nb, str element_name, obibool_t value):
-        if obi_column_set_obibool_with_elt_name(self.pointer, line_nb, str2bytes(element_name), value) < 0 :
+    cpdef set_item(self, index_t line_nb, str element_name, object value):
+        if value is None :
+            value = OBIBool_NA
+        if obi_column_set_obibool_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name), <obibool_t> value) < 0:
             raise Exception("Problem setting a value in a column")
- 
-    cpdef close(self):
-        if obi_truncate_and_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
-     

python/obitools3/obidms/_obidmscolumn_char.cfiles

@@ -1,16 +1,51 @@
-../../../src/obidmscolumn_char.c
-../../../src/obidmscolumn_char.h
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
 ../../../src/obidmscolumndir.c
-../../../src/obidms.h
-../../../src/obidms.c
 ../../../src/obierrno.h
 ../../../src/obierrno.c
 ../../../src/obilittlebigman.h
 ../../../src/obilittlebigman.c
 ../../../src/obitypes.h
 ../../../src/obitypes.c
-../../../src/private_at_functions.h
-../../../src/private_at_functions.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/utils.h
+../../../src/utils.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h

python/obitools3/obidms/_obidmscolumn_char.pxd

@@ -1,17 +1,14 @@
 #cython: language_level=3
 
-from .capi.obitypes cimport obichar_t
-from .capi.obidmscolumn cimport OBIDMS_column_p
-                                                
-from ._obidms cimport OBIDMS_column
+from .capi.obitypes cimport index_t
+from ._obidms       cimport OBIDMS_column, OBIDMS_column_multi_elts
 
 
 cdef class OBIDMS_column_char(OBIDMS_column):
-    cpdef object get_item(self, size_t line_nb, str element_name)
-    cpdef set_item(self, size_t line_nb, str element_name, bytes value)
-    cpdef close(self)
-
-cdef class OBIDMS_column_char_writable(OBIDMS_column_char):
-    cpdef set_item(self, size_t line_nb, str element_name, bytes value)
-    cpdef close(self)
+    cpdef object get_line(self, index_t line_nb)
+    cpdef set_line(self, index_t line_nb, object value)
 
+cdef class OBIDMS_column_multi_elts_char(OBIDMS_column_multi_elts):
+    cpdef object get_item(self, index_t line_nb, str element_name)
+    cpdef object get_line(self, index_t line_nb)
+    cpdef set_item(self, index_t line_nb, str element_name, object value)

python/obitools3/obidms/_obidmscolumn_char.pyx

@@ -1,44 +1,76 @@
 #cython: language_level=3
  
-from .capi.obidmscolumn cimport obi_close_column,\
-                                obi_truncate_and_close_column, \
-                                obi_column_get_obichar_with_elt_name, \
-                                obi_column_set_obichar_with_elt_name
+from .capi.obiview      cimport obi_column_get_obichar_with_elt_name_in_view, \
+                                obi_column_get_obichar_with_elt_idx_in_view, \
+                                obi_column_set_obichar_with_elt_name_in_view, \
+                                obi_column_set_obichar_with_elt_idx_in_view
 from .capi.obierrno     cimport obi_errno
-from .capi.obitypes     cimport OBIChar_NA
+from .capi.obitypes     cimport OBIChar_NA, obichar_t
 
-from obitools3.utils cimport str2bytes
+from obitools3.utils cimport str2bytes, bytes2str
 
 
-cdef class OBIDMS_column_char(OBIDMS_column) :
- 
-    cpdef object get_item(self, size_t line_nb, str element_name):
-        cdef char value
+cdef class OBIDMS_column_char(OBIDMS_column):
+
+    cpdef object get_line(self, index_t line_nb):
+        cdef obichar_t value
         cdef object result
-        value = obi_column_get_obichar_with_elt_name(self.pointer, line_nb, str2bytes(element_name))
+        value = obi_column_get_obichar_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0)
+        if obi_errno > 0 :
+            raise IndexError(line_nb)
+        if value == OBIChar_NA :
+            result = None
+        else :
+            result = bytes2str(value)
+        return result
+
+    cpdef set_line(self, index_t line_nb, object value):
+        if value is None :
+            value = OBIChar_NA
+        if obi_column_set_obichar_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0, str2bytes(value)[0]) < 0:
+            raise Exception("Problem setting a value in a column")
+
+
+cdef class OBIDMS_column_multi_elts_char(OBIDMS_column_multi_elts):
+
+    cpdef object get_item(self, index_t line_nb, str element_name):
+        cdef obichar_t value
+        cdef object result
+        value = obi_column_get_obichar_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name))
         if obi_errno > 0 :
             raise IndexError(line_nb, element_name)
         if value == OBIChar_NA :
             result = None
         else :
-            result = <bytes> value
+            result = bytes2str(value)
         return result
 
-    cpdef set_item(self, size_t line_nb, str element_name, bytes value):
-        raise Exception("Column is read-only")
- 
-    cpdef close(self):
-        if obi_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
- 
-         
-cdef class OBIDMS_column_char_writable(OBIDMS_column_char) :     
- 
-    cpdef set_item(self, size_t line_nb, str element_name, bytes value):
-        if obi_column_set_obichar_with_elt_name(self.pointer, line_nb, str2bytes(element_name), value[0]) < 0:
+    cpdef object get_line(self, index_t line_nb) :
+        cdef obichar_t value
+        cdef object value_in_result
+        cdef dict result
+        cdef index_t i
+        cdef bint all_NA
+        result = {}
+        all_NA = True
+        for i in range(self.nb_elements_per_line) :
+            value = obi_column_get_obichar_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, i)
+            if obi_errno > 0 :
+                raise IndexError(line_nb)
+            if value == OBIChar_NA :
+                value_in_result = None
+            else :
+                value_in_result = bytes2str(value)   
+            result[self.elements_names[i]] = value_in_result
+            if all_NA and (value_in_result is not None) :
+                all_NA = False
+        if all_NA :
+            result = None
+        return result
+
+    cpdef set_item(self, index_t line_nb, str element_name, object value):
+        if value is None :
+            value = OBIChar_NA
+        if obi_column_set_obichar_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name), str2bytes(value)[0]) < 0:
             raise Exception("Problem setting a value in a column")
- 
-    cpdef close(self):
-        if obi_truncate_and_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
-      
+

python/obitools3/obidms/_obidmscolumn_float.cfiles

@@ -1,16 +1,51 @@
-../../../src/obidmscolumn_float.c
-../../../src/obidmscolumn_float.h
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
 ../../../src/obidmscolumndir.c
-../../../src/obidms.h
-../../../src/obidms.c
 ../../../src/obierrno.h
 ../../../src/obierrno.c
 ../../../src/obilittlebigman.h
 ../../../src/obilittlebigman.c
 ../../../src/obitypes.h
 ../../../src/obitypes.c
-../../../src/private_at_functions.h
-../../../src/private_at_functions.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/utils.h
+../../../src/utils.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h

python/obitools3/obidms/_obidmscolumn_float.pxd

@@ -1,16 +1,14 @@
 #cython: language_level=3
 
-from .capi.obitypes cimport obifloat_t
-from .capi.obidmscolumn cimport OBIDMS_column_p
-                                                
-from ._obidms cimport OBIDMS_column
+from .capi.obitypes cimport index_t
+from ._obidms       cimport OBIDMS_column, OBIDMS_column_multi_elts
 
 
 cdef class OBIDMS_column_float(OBIDMS_column):
-    cpdef object get_item(self, size_t line_nb, str element_name)
-    cpdef set_item(self, size_t line_nb, str element_name, obifloat_t value)
-    cpdef close(self)
-    
-cdef class OBIDMS_column_float_writable(OBIDMS_column_float):
-    cpdef set_item(self, size_t line_nb, str element_name, obifloat_t value)
-    cpdef close(self)
+    cpdef object get_line(self, index_t line_nb)
+    cpdef set_line(self, index_t line_nb, object value)
+
+cdef class OBIDMS_column_multi_elts_float(OBIDMS_column_multi_elts):
+    cpdef object get_item(self, index_t line_nb, str element_name)
+    cpdef object get_line(self, index_t line_nb)
+    cpdef set_item(self, index_t line_nb, str element_name, object value)

python/obitools3/obidms/_obidmscolumn_float.pyx

@@ -1,44 +1,76 @@
 #cython: language_level=3
 
-from .capi.obidmscolumn cimport obi_close_column,\
-                                obi_truncate_and_close_column, \
-                                obi_column_get_obifloat_with_elt_name, \
-                                obi_column_set_obifloat_with_elt_name
+from .capi.obiview      cimport obi_column_get_obifloat_with_elt_name_in_view, \
+                                obi_column_get_obifloat_with_elt_idx_in_view, \
+                                obi_column_set_obifloat_with_elt_name_in_view, \
+                                obi_column_set_obifloat_with_elt_idx_in_view
 from .capi.obierrno     cimport obi_errno
-from .capi.obitypes     cimport OBIFloat_NA
+from .capi.obitypes     cimport OBIFloat_NA, obifloat_t
 
 from obitools3.utils cimport str2bytes
 
 
 cdef class OBIDMS_column_float(OBIDMS_column):
 
-    cpdef object get_item(self, size_t line_nb, str element_name):
+    cpdef object get_line(self, index_t line_nb):
         cdef obifloat_t value
         cdef object result
-        value = obi_column_get_obifloat_with_elt_name(self.pointer, line_nb, str2bytes(element_name))
+        value = obi_column_get_obifloat_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0)
+        if obi_errno > 0 :
+            raise IndexError(line_nb)
+        if value == OBIFloat_NA :
+            result = None
+        else :
+            result = <double> value
+        return result
+
+    cpdef set_line(self, index_t line_nb, object value):
+        if value is None :
+            value = OBIFloat_NA
+        if obi_column_set_obifloat_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0, <obifloat_t> value) < 0:
+            raise Exception("Problem setting a value in a column")
+
+
+cdef class OBIDMS_column_multi_elts_float(OBIDMS_column_multi_elts):
+
+    cpdef object get_item(self, index_t line_nb, str element_name):
+        cdef obifloat_t value
+        cdef object result
+        value = obi_column_get_obifloat_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name))
         if obi_errno > 0 :
             raise IndexError(line_nb, element_name)
         if value == OBIFloat_NA :
             result = None
         else :
-            result = <float> value
+            result = <double> value
         return result
-    
-    cpdef set_item(self, size_t line_nb, str element_name, obifloat_t value):
-        raise Exception("Column is read-only")
 
-    cpdef close(self):
-        if obi_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
-                  
-
-cdef class OBIDMS_column_float_writable(OBIDMS_column_float):     
-
-    cpdef set_item(self, size_t line_nb, str element_name, obifloat_t value):
-        if obi_column_set_obifloat_with_elt_name(self.pointer, line_nb, str2bytes(element_name), value) < 0:
+    cpdef object get_line(self, index_t line_nb) :
+        cdef obifloat_t value
+        cdef object value_in_result
+        cdef dict result
+        cdef index_t i
+        cdef bint all_NA
+        result = {}
+        all_NA = True
+        for i in range(self.nb_elements_per_line) :
+            value = obi_column_get_obifloat_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, i)
+            if obi_errno > 0 :
+                raise IndexError(line_nb)
+            if value == OBIFloat_NA :
+                value_in_result = None
+            else :
+                value_in_result = <double> value
+            result[self.elements_names[i]] = value_in_result
+            if all_NA and (value_in_result is not None) :
+                all_NA = False
+        if all_NA :
+            result = None
+        return result
+   
+    cpdef set_item(self, index_t line_nb, str element_name, object value):
+        if value is None :
+            value = OBIFloat_NA
+        if obi_column_set_obifloat_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name), <obifloat_t> value) < 0:
             raise Exception("Problem setting a value in a column")
-
-    cpdef close(self):
-        if obi_truncate_and_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
  

python/obitools3/obidms/_obidmscolumn_idx.cfiles

@@ -1,16 +0,0 @@
-../../../src/obidmscolumn_idx.c
-../../../src/obidmscolumn_idx.h
-../../../src/obidmscolumn.h
-../../../src/obidmscolumn.c
-../../../src/obidmscolumndir.h
-../../../src/obidmscolumndir.c
-../../../src/obidms.h
-../../../src/obidms.c
-../../../src/obierrno.h
-../../../src/obierrno.c
-../../../src/obilittlebigman.h
-../../../src/obilittlebigman.c
-../../../src/obitypes.h
-../../../src/obitypes.c
-../../../src/private_at_functions.h
-../../../src/private_at_functions.c

python/obitools3/obidms/_obidmscolumn_idx.pxd

@@ -1,16 +0,0 @@
-#cython: language_level=3
-
-from .capi.obitypes cimport obiidx_t
-from .capi.obidmscolumn cimport OBIDMS_column_p
-                                                
-from ._obidms cimport OBIDMS_column
-
-
-cdef class OBIDMS_column_idx(OBIDMS_column):
-    cpdef object get_item(self, size_t line_nb, str element_name)
-    cpdef set_item(self, size_t line_nb, str element_name, obiidx_t value)
-    cpdef close(self)
-    
-cdef class OBIDMS_column_idx_writable(OBIDMS_column_idx):
-    cpdef set_item(self, size_t line_nb, str element_name, obiidx_t value)
-    cpdef close(self)

python/obitools3/obidms/_obidmscolumn_idx.pyx

@@ -1,46 +0,0 @@
-#cython: language_level=3
-
-from .capi.obidmscolumn cimport obi_close_column,\
-                                obi_truncate_and_close_column, \
-                                obi_column_get_obiidx_with_elt_name, \
-                                obi_column_set_obiidx_with_elt_name
-from .capi.obierrno     cimport obi_errno
-from .capi.obitypes     cimport OBIIdx_NA
-
-from obitools3.utils cimport str2bytes
-
-from cpython.int cimport PyInt_FromSsize_t
-
-
-cdef class OBIDMS_column_idx(OBIDMS_column):
-
-    cpdef object get_item(self, size_t line_nb, str element_name):
-        cdef obiidx_t value
-        cdef object result
-        value = obi_column_get_obiidx_with_elt_name(self.pointer, line_nb, str2bytes(element_name))
-        if obi_errno > 0 :
-            raise IndexError(line_nb, element_name)
-        if value == OBIIdx_NA :
-            result = None
-        else :
-            result = PyInt_FromSsize_t(value)
-        return result
-    
-    cpdef set_item(self, size_t line_nb, str element_name, obiidx_t value):
-        raise Exception("Column is read-only")
-
-    cpdef close(self):
-        if obi_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
-                  
-
-cdef class OBIDMS_column_idx_writable(OBIDMS_column_idx):     
-
-    cpdef set_item(self, size_t line_nb, str element_name, obiidx_t value):
-        if obi_column_set_obiidx_with_elt_name(self.pointer, line_nb, str2bytes(element_name), value) < 0:
-            raise Exception("Problem setting a value in a column")
-
-    cpdef close(self):
-        if obi_truncate_and_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
- 

python/obitools3/obidms/_obidmscolumn_int.cfiles

@@ -1,16 +1,51 @@
-../../../src/obidmscolumn_int.c
-../../../src/obidmscolumn_int.h
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
 ../../../src/obidmscolumndir.c
-../../../src/obidms.h
-../../../src/obidms.c
 ../../../src/obierrno.h
 ../../../src/obierrno.c
 ../../../src/obilittlebigman.h
 ../../../src/obilittlebigman.c
 ../../../src/obitypes.h
 ../../../src/obitypes.c
-../../../src/private_at_functions.h
-../../../src/private_at_functions.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/utils.h
+../../../src/utils.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h

python/obitools3/obidms/_obidmscolumn_int.pxd

@@ -1,16 +1,14 @@
 #cython: language_level=3
 
-from .capi.obitypes cimport obiint_t
-from .capi.obidmscolumn cimport OBIDMS_column_p
-                                                
-from ._obidms cimport OBIDMS_column
+from .capi.obitypes cimport index_t
+from ._obidms       cimport OBIDMS_column, OBIDMS_column_multi_elts
 
 
 cdef class OBIDMS_column_int(OBIDMS_column):
-    cpdef object get_item(self, size_t line_nb, str element_name)
-    cpdef set_item(self, size_t line_nb, str element_name, obiint_t value)
-    cpdef close(self)
-    
-cdef class OBIDMS_column_int_writable(OBIDMS_column_int):
-    cpdef set_item(self, size_t line_nb, str element_name, obiint_t value)
-    cpdef close(self)
+    cpdef object get_line(self, index_t line_nb)
+    cpdef set_line(self, index_t line_nb, object value)
+
+cdef class OBIDMS_column_multi_elts_int(OBIDMS_column_multi_elts):
+    cpdef object get_item(self, index_t line_nb, str element_name)
+    cpdef object get_line(self, index_t line_nb)
+    cpdef set_item(self, index_t line_nb, str element_name, object value)

python/obitools3/obidms/_obidmscolumn_int.pyx

@@ -1,25 +1,44 @@
 #cython: language_level=3
 
-from .capi.obidmscolumn cimport obi_close_column,\
-                                obi_truncate_and_close_column, \
-                                obi_column_get_obiint_with_elt_name, \
-                                obi_column_set_obiint_with_elt_name
+from .capi.obiview      cimport obi_column_get_obiint_with_elt_name_in_view, \
+                                obi_column_get_obiint_with_elt_idx_in_view, \
+                                obi_column_set_obiint_with_elt_name_in_view, \
+                                obi_column_set_obiint_with_elt_idx_in_view
 from .capi.obierrno     cimport obi_errno
-from .capi.obitypes     cimport OBIInt_NA
+from .capi.obitypes     cimport OBIInt_NA, obiint_t
 
 from obitools3.utils cimport str2bytes
 
 from cpython.int cimport PyInt_FromLong
 
-from ._obidms cimport OBIDMS_column
-
 
 cdef class OBIDMS_column_int(OBIDMS_column):
 
-    cpdef object get_item(self, size_t line_nb, str element_name):
+    cpdef object get_line(self, index_t line_nb):
         cdef obiint_t value
         cdef object result
-        value = obi_column_get_obiint_with_elt_name(self.pointer, line_nb, str2bytes(element_name))
+        value = obi_column_get_obiint_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0)
+        if obi_errno > 0 :
+            raise IndexError(line_nb)
+        if value == OBIInt_NA :
+            result = None
+        else :
+            result = PyInt_FromLong(value)
+        return result
+            
+    cpdef set_line(self, index_t line_nb, object value):
+        if value is None :
+            value = OBIInt_NA
+        if obi_column_set_obiint_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0, <obiint_t> value) < 0:
+            raise Exception("Problem setting a value in a column")
+
+
+cdef class OBIDMS_column_multi_elts_int(OBIDMS_column_multi_elts):
+
+    cpdef object get_item(self, index_t line_nb, str element_name):
+        cdef obiint_t value
+        cdef object result
+        value = obi_column_get_obiint_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name))
         if obi_errno > 0 :
             raise IndexError(line_nb, element_name)
         if value == OBIInt_NA :
@@ -28,21 +47,32 @@ cdef class OBIDMS_column_int(OBIDMS_column):
             result = PyInt_FromLong(value)
         return result
     
-    cpdef set_item(self, size_t line_nb, str element_name, obiint_t value):
-        raise Exception("Column is read-only")
+    cpdef object get_line(self, index_t line_nb) :
+        cdef obiint_t value
+        cdef object value_in_result
+        cdef dict result
+        cdef index_t i
+        cdef bint all_NA
+        result = {}
+        all_NA = True
+        for i in range(self.nb_elements_per_line) :
+            value = obi_column_get_obiint_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, i)
+            if obi_errno > 0 :
+                raise IndexError(line_nb)
+            if value == OBIInt_NA :
+                value_in_result = None
+            else :
+                value_in_result = PyInt_FromLong(value)
+            result[self.elements_names[i]] = value_in_result
+            if all_NA and (value_in_result is not None) :
+                all_NA = False
+        if all_NA :
+            result = None   # TODO discuss
+        return result
 
-    cpdef close(self):
-        if obi_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
-                  
-
-cdef class OBIDMS_column_int_writable(OBIDMS_column_int):     
-
-    cpdef set_item(self, size_t line_nb, str element_name, obiint_t value):
-        if obi_column_set_obiint_with_elt_name(self.pointer, line_nb, str2bytes(element_name), value) < 0:
+    cpdef set_item(self, index_t line_nb, str element_name, object value):
+        if value is None :
+            value = OBIInt_NA
+        if obi_column_set_obiint_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name), <obiint_t> value) < 0:
             raise Exception("Problem setting a value in a column")
 
-    cpdef close(self):
-        if obi_truncate_and_close_column(self.pointer) < 0 :
-            raise Exception("Problem closing a column")
- 

python/obitools3/obidms/_obidmscolumn_seq.cfiles

@@ -0,0 +1,51 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn.h
+../../../src/obidmscolumn.c
+../../../src/obidmscolumndir.h
+../../../src/obidmscolumndir.c
+../../../src/obierrno.h
+../../../src/obierrno.c
+../../../src/obilittlebigman.h
+../../../src/obilittlebigman.c
+../../../src/obitypes.h
+../../../src/obitypes.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/utils.h
+../../../src/utils.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h

python/obitools3/obidms/_obidmscolumn_seq.pxd

@@ -0,0 +1,14 @@
+#cython: language_level=3
+
+from .capi.obitypes cimport index_t
+from ._obidms       cimport OBIDMS_column, OBIDMS_column_multi_elts
+
+
+cdef class OBIDMS_column_seq(OBIDMS_column):
+    cpdef object get_line(self, index_t line_nb)
+    cpdef set_line(self, index_t line_nb, object value)
+
+cdef class OBIDMS_column_multi_elts_seq(OBIDMS_column_multi_elts):
+    cpdef object get_item(self, index_t line_nb, str element_name)
+    cpdef object get_line(self, index_t line_nb)
+    cpdef set_item(self, index_t line_nb, str element_name, object value)

python/obitools3/obidms/_obidmscolumn_seq.pyx

@@ -0,0 +1,88 @@
+#cython: language_level=3
+ 
+from .capi.obiview      cimport obi_column_get_obiseq_with_elt_name_in_view, \
+                                obi_column_get_obiseq_with_elt_idx_in_view, \
+                                obi_column_set_obiseq_with_elt_name_in_view, \
+                                obi_column_set_obiseq_with_elt_idx_in_view
+from .capi.obierrno     cimport obi_errno
+from .capi.obitypes     cimport OBISeq_NA, const_char_p
+
+from obitools3.utils cimport str2bytes, bytes2str
+
+from libc.stdlib cimport free
+from libc.string cimport strcmp
+
+
+cdef class OBIDMS_column_seq(OBIDMS_column):
+
+    cpdef object get_line(self, index_t line_nb):
+        cdef char*  value
+        cdef object result
+        value = obi_column_get_obiseq_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0)
+        if obi_errno > 0 :
+            raise IndexError(line_nb)
+        if strcmp(value, OBISeq_NA) == 0 :
+            result = None
+        else :
+            result = bytes2str(value)
+            free(value)
+        return result
+
+    cpdef set_line(self, index_t line_nb, object value):
+        cdef bytes value_b
+        if value is None :
+            value_b = OBISeq_NA
+        else :
+            value_b = str2bytes(value)
+        if obi_column_set_obiseq_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0, value_b) < 0:
+            raise Exception("Problem setting a value in a column")
+
+
+cdef class OBIDMS_column_multi_elts_seq(OBIDMS_column_multi_elts):
+
+    cpdef object get_item(self, index_t line_nb, str element_name):
+        cdef char*  value
+        cdef object result
+        value = obi_column_get_obiseq_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name))
+        if obi_errno > 0 :
+            raise IndexError(line_nb, element_name)
+        if strcmp(value, OBISeq_NA) == 0 :
+            result = None
+        else :
+            result = bytes2str(value)
+            free(value)
+        return result
+
+    cpdef object get_line(self, index_t line_nb) :
+        cdef char*   value
+        cdef object  value_in_result
+        cdef dict    result
+        cdef index_t i
+        cdef bint    all_NA
+        result = {}
+        all_NA = True
+        for i in range(self.nb_elements_per_line) :
+            value = obi_column_get_obiseq_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, i)
+            if obi_errno > 0 :
+                raise IndexError(line_nb)
+            if strcmp(value, OBISeq_NA) == 0 :
+                value_in_result = None
+            else :
+                value_in_result = bytes2str(value)
+                free(value)    
+            result[self.elements_names[i]] = value_in_result
+            if all_NA and (value_in_result is not None) :
+                all_NA = False
+        if all_NA :
+            result = None
+        return result
+
+    cpdef set_item(self, index_t line_nb, str element_name, object value):
+        cdef bytes value_b
+        if value is None :
+            value_b = OBISeq_NA
+        else :
+            value_b = str2bytes(value)
+        if obi_column_set_obiseq_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name), value_b) < 0:
+            raise Exception("Problem setting a value in a column")
+    

python/obitools3/obidms/_obidmscolumn_str.cfiles

@@ -0,0 +1,51 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn.h
+../../../src/obidmscolumn.c
+../../../src/obidmscolumndir.h
+../../../src/obidmscolumndir.c
+../../../src/obierrno.h
+../../../src/obierrno.c
+../../../src/obilittlebigman.h
+../../../src/obilittlebigman.c
+../../../src/obitypes.h
+../../../src/obitypes.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/utils.h
+../../../src/utils.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h

python/obitools3/obidms/_obidmscolumn_str.pxd

@@ -0,0 +1,14 @@
+#cython: language_level=3
+
+from .capi.obitypes cimport index_t
+from ._obidms       cimport OBIDMS_column, OBIDMS_column_multi_elts
+
+
+cdef class OBIDMS_column_str(OBIDMS_column):
+    cpdef object get_line(self, index_t line_nb)
+    cpdef set_line(self, index_t line_nb, object value)
+
+cdef class OBIDMS_column_multi_elts_str(OBIDMS_column_multi_elts):
+    cpdef object get_item(self, index_t line_nb, str element_name)
+    cpdef object get_line(self, index_t line_nb)
+    cpdef set_item(self, index_t line_nb, str element_name, object value)

python/obitools3/obidms/_obidmscolumn_str.pyx

@@ -0,0 +1,87 @@
+#cython: language_level=3
+ 
+from .capi.obiview      cimport obi_column_get_obistr_with_elt_name_in_view, \
+                                obi_column_get_obistr_with_elt_idx_in_view, \
+                                obi_column_set_obistr_with_elt_name_in_view, \
+                                obi_column_set_obistr_with_elt_idx_in_view
+from .capi.obierrno     cimport obi_errno
+from .capi.obitypes     cimport OBIStr_NA, const_char_p
+
+from obitools3.utils cimport str2bytes, bytes2str
+
+from libc.string cimport strcmp
+
+
+cdef class OBIDMS_column_str(OBIDMS_column):
+
+    cpdef object get_line(self, index_t line_nb):
+        cdef const_char_p value
+        cdef object       result
+        value = obi_column_get_obistr_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0)
+        if obi_errno > 0 :
+            raise IndexError(line_nb)
+        if strcmp(value, OBIStr_NA) == 0 :
+            result = None
+        else :
+            result = bytes2str(value)
+        # NOTE: value is not freed because the pointer points to a mmapped region in an AVL data file. (TODO discuss)
+        return result
+
+    cpdef set_line(self, index_t line_nb, object value):
+        cdef bytes value_b
+        if value is None :
+            value_b = OBIStr_NA
+        else :
+            value_b = str2bytes(value)
+        if obi_column_set_obistr_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0, value_b) < 0:
+            raise Exception("Problem setting a value in a column")
+
+
+cdef class OBIDMS_column_multi_elts_str(OBIDMS_column_multi_elts):
+
+    cpdef object get_item(self, index_t line_nb, str element_name):
+        cdef const_char_p value
+        cdef object       result
+        value = obi_column_get_obistr_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name))
+        if obi_errno > 0 :
+            raise IndexError(line_nb, element_name)
+        if strcmp(value, OBIStr_NA) == 0 :
+            result = None
+        else :
+            result = bytes2str(value)
+        # NOTE: value is not freed because the pointer points to a mmapped region in an AVL data file. (TODO discuss)
+        return result
+
+    cpdef object get_line(self, index_t line_nb) :
+        cdef const_char_p value
+        cdef object       value_in_result
+        cdef dict         result
+        cdef index_t      i
+        cdef bint         all_NA
+        result = {}
+        all_NA = True
+        for i in range(self.nb_elements_per_line) :
+            value = obi_column_get_obistr_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, i)
+            if obi_errno > 0 :
+                raise IndexError(line_nb)
+            if strcmp(value, OBIStr_NA) == 0 :
+                value_in_result = None
+            else :
+                value_in_result = bytes2str(value)
+            # NOTE: value is not freed because the pointer points to a mmapped region in an AVL data file. (TODO discuss)
+            result[self.elements_names[i]] = value_in_result
+            if all_NA and (value_in_result is not None) :
+                all_NA = False
+        if all_NA :
+            result = None
+        return result
+
+    cpdef set_item(self, index_t line_nb, str element_name, object value):
+        cdef bytes value_b
+        if value is None :
+            value_b = OBIStr_NA
+        else :
+            value_b = str2bytes(value)
+        if obi_column_set_obistr_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name), value_b) < 0:
+            raise Exception("Problem setting a value in a column")
+    

python/obitools3/obidms/_obiseq.cfiles

@@ -0,0 +1,51 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn.h
+../../../src/obidmscolumn.c
+../../../src/obidmscolumndir.h
+../../../src/obidmscolumndir.c
+../../../src/obierrno.h
+../../../src/obierrno.c
+../../../src/obilittlebigman.h
+../../../src/obilittlebigman.c
+../../../src/obitypes.h
+../../../src/obitypes.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/utils.h
+../../../src/utils.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h

python/obitools3/obidms/_obiseq.pxd

@@ -0,0 +1,30 @@
+#cython: language_level=3
+
+from ._obidms cimport OBIView_line
+
+
+cdef class OBI_Seq(dict) :
+    cdef str id
+    cdef str definition
+    cdef str sequence
+
+    cpdef set_id(self, str id)
+    cpdef get_id(self)
+    cpdef set_definition(self, str definition)
+    cpdef get_definition(self)
+    cpdef get_sequence(self)
+
+
+cdef class OBI_Nuc_Seq(OBI_Seq) :
+    #cpdef str reverse_complement(self)    
+    cpdef set_sequence(self, str sequence)
+
+
+cdef class OBI_Nuc_Seq_Stored(OBIView_line) :
+    cpdef set_id(self, str id)
+    cpdef get_id(self)
+    cpdef set_definition(self, str definition)
+    cpdef get_definition(self)
+    cpdef set_sequence(self, str sequence)
+    cpdef get_sequence(self)
+   # cpdef str reverse_complement(self)

python/obitools3/obidms/_obiseq.pyx

@@ -0,0 +1,75 @@
+#cython: language_level=3
+
+from obitools3.utils cimport bytes2str, str2bytes
+
+from .capi.obiview cimport NUC_SEQUENCE_COLUMN, \
+                           ID_COLUMN, \
+                           DEFINITION_COLUMN
+
+cdef class OBI_Seq(dict) :
+
+    def __init__(self, str id, str seq, str definition=None) :
+        self.set_id(id)
+        self.set_sequence(seq)
+        if definition is not None :
+            self.set_definition(definition)
+        
+    cpdef set_id(self, str id) :
+        self.id = id
+        self[bytes2str(ID_COLUMN)] = id
+        
+    cpdef get_id(self) :
+        return self.id
+            
+    cpdef set_definition(self, str definition) :
+        self.definition = definition
+        self[bytes2str(DEFINITION_COLUMN)] = definition
+        
+    cpdef get_definition(self) :
+        return self.definition
+        
+    cpdef get_sequence(self) :
+        return self.sequence
+
+    def __str__(self) :
+        return self.sequence    # or not
+
+
+cdef class OBI_Nuc_Seq(OBI_Seq) :
+    
+    cpdef set_sequence(self, str sequence) :
+        self.sequence = sequence
+        self[bytes2str(NUC_SEQUENCE_COLUMN)] = sequence
+
+#    cpdef str reverse_complement(self) :    TODO in C ?
+#        pass
+
+
+cdef class OBI_Nuc_Seq_Stored(OBIView_line) :
+
+    cpdef set_id(self, str id) :
+        self[bytes2str(ID_COLUMN)] = id
+        
+    cpdef get_id(self) :
+        return self[bytes2str(ID_COLUMN)]
+            
+    cpdef set_definition(self, str definition) :
+        self[bytes2str(DEFINITION_COLUMN)] = definition
+
+    cpdef get_definition(self) :
+        return self[bytes2str(DEFINITION_COLUMN)]
+
+    cpdef set_sequence(self, str sequence) :
+        self[bytes2str(NUC_SEQUENCE_COLUMN)] = sequence
+
+    cpdef get_sequence(self) :
+        return self[bytes2str(NUC_SEQUENCE_COLUMN)]
+    
+#    def __str__(self) :
+#        return self[bytes2str(NUC_SEQUENCE_COLUMN)]    # or not
+    
+#    cpdef str reverse_complement(self) :    TODO in C ?
+#        pass
+
+    # TODO static method to import?
+    

python/obitools3/obidms/_obitaxo.cfiles

@@ -0,0 +1,51 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn.h
+../../../src/obidmscolumn.c
+../../../src/obidmscolumndir.h
+../../../src/obidmscolumndir.c
+../../../src/obierrno.h
+../../../src/obierrno.c
+../../../src/obilittlebigman.h
+../../../src/obilittlebigman.c
+../../../src/obitypes.h
+../../../src/obitypes.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/utils.h
+../../../src/utils.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h

python/obitools3/obidms/_obitaxo.pxd

@@ -0,0 +1,31 @@
+#cython: language_level=3
+
+from .capi.obitaxonomy cimport ecotx_t, OBIDMS_taxonomy_p
+
+from libc.stdint   cimport int32_t
+
+
+cdef class OBI_Taxonomy :
+
+    cdef str name
+    cdef OBIDMS_taxonomy_p pointer
+    
+    cpdef close(self)
+
+ 
+cdef class OBI_Taxon :
+
+    cdef ecotx_t* pointer
+    cdef int32_t  taxid
+    cdef int32_t  rank
+    cdef int32_t  farest
+    cdef ecotx_t* parent
+    cdef str name
+        
+    cpdef int32_t taxid(self)
+    
+    cpdef int32_t rank(self)
+    
+    cpdef int32_t farest(self)
+    
+    cpdef OBI_Taxon parent(self)

python/obitools3/obidms/_obitaxo.pyx

@@ -0,0 +1,65 @@
+#cython: language_level=3
+
+from obitools3.utils   cimport bytes2str, str2bytes
+
+from .capi.obitaxonomy cimport obi_read_taxonomy, \
+                               obi_close_taxonomy, \
+                               obi_taxo_get_taxon_with_taxid
+
+from ._obidms cimport OBIDMS
+
+from cpython.pycapsule cimport PyCapsule_New, PyCapsule_GetPointer
+
+
+cdef class OBI_Taxonomy :
+
+    def __init__(self, OBIDMS dms, str name) :
+        
+        self.name = name
+        self.pointer = obi_read_taxonomy(dms.pointer, str2bytes(name), True)    # TODO discuss
+    
+    
+    def __getitem__(self, object ref):
+        
+        cdef ecotx_t* taxon_p
+        cdef object taxon_capsule
+        
+        if type(ref) == int :
+            taxon_p = obi_taxo_get_taxon_with_taxid(self.pointer, ref)
+            taxon_capsule = PyCapsule_New(taxon_p, NULL, NULL)
+            return OBI_Taxon(taxon_capsule)
+    
+         
+    cpdef close(self) :
+        if (obi_close_taxonomy(self.pointer) < 0) :
+            raise Exception("Error closing the taxonomy")
+        
+
+cdef class OBI_Taxon :    # dict subclass?
+
+    def __init__(self, object taxon_capsule) :
+        
+        cdef ecotx_t* taxon
+        
+        taxon = <ecotx_t*> PyCapsule_GetPointer(taxon_capsule, NULL)   
+        self.pointer = taxon
+        self.taxid = taxon.taxid
+        self.rank = taxon.rank
+        self.farest = taxon.farest
+        self.parent = taxon.parent
+        self.name = bytes2str(taxon.name)
+        
+    cpdef int32_t taxid(self):
+        return self.taxid
+    
+    cpdef int32_t rank(self):
+        return self.rank
+    
+    cpdef int32_t farest(self):
+        return self.farest
+    
+    cpdef OBI_Taxon parent(self):
+        cdef object parent_capsule
+        parent_capsule = PyCapsule_New(self.parent, NULL, NULL)
+        return OBI_Taxon(parent_capsule)
+    

python/obitools3/obidms/capi/obidmscolumn.pxd

@@ -8,106 +8,189 @@ from ..capi.obitypes cimport const_char_p, \
                              obibool_t, \
                              obichar_t, \
                              obifloat_t, \
-                             obiidx_t
+                             index_t, \
+                             time_t
+
 
 cdef extern from "obidmscolumn.h" nogil:
-    struct OBIDMS_column_t:
-        pass
+    
+    struct OBIDMS_column_header_t:
+        size_t          header_size
+        size_t          data_size
+        index_t         line_count
+        index_t         lines_used
+        index_t         nb_elements_per_line
+        const_char_p    elements_names
+        OBIType_t       returned_data_type
+        OBIType_t       stored_data_type
+        time_t          creation_date
+        obiversion_t    version
+        obiversion_t    cloned_from
+        const_char_p    name
+        const_char_p    indexer_name
+        const_char_p    comments
+    
+    ctypedef OBIDMS_column_header_t* OBIDMS_column_header_p
 
+    struct OBIDMS_column_t:
+        OBIDMS_p               dms
+        OBIDMS_column_header_p header
+        bint                   writable
+        
     ctypedef OBIDMS_column_t* OBIDMS_column_p
 
     OBIDMS_column_p obi_create_column(OBIDMS_p dms, 
                                       const_char_p column_name, 
                                       OBIType_t type, 
-                                      size_t nb_lines, 
-                                      size_t nb_elements_per_line, 
-                                      const_char_p elements_names)
-    
-    size_t obi_column_get_nb_lines_used(OBIDMS_column_p column)
-    
-    const_char_p obi_column_get_elements_names(OBIDMS_column_p column)
-    
-    void obi_column_make_unwritable(OBIDMS_column_p column)
-    
+                                      index_t nb_lines, 
+                                      index_t nb_elements_per_line, 
+                                      const_char_p elements_names,
+                                      const_char_p indexer_name,
+                                      const_char_p comments)
+                
     OBIDMS_column_p obi_open_column(OBIDMS_p dms, 
                                     const_char_p column_name, 
                                     obiversion_t version_number)
-    
-    OBIType_t obi_column_get_type(OBIDMS_column_p column)
-    
+        
     int obi_close_column(OBIDMS_column_p column)
-    
-    OBIType_t obi_column_get_data_type_from_name(OBIDMS_p dms, 
-                                                 const_char_p column_name)
-    
+        
     OBIDMS_column_p obi_clone_column(OBIDMS_p dms, 
+                                     OBIDMS_column_p line_selection,
                                      const_char_p column_name, 
                                      obiversion_t version_number, 
                                      bint clone_data)
     
-    int obi_truncate_and_close_column(OBIDMS_column_p column)
+    int obi_close_column(OBIDMS_column_p column)
     
     obiversion_t obi_column_get_latest_version_from_name(OBIDMS_p dms, 
                                                          const_char_p column_name)
+        
+    OBIDMS_column_header_p obi_column_get_header_from_name(OBIDMS_p dms, 
+                                                           const_char_p column_name,
+                                                           obiversion_t version_number)
     
-    OBIType_t obi_column_get_data_type_from_name(OBIDMS_p dms, 
-                                                 const_char_p column_name)
-    
-    size_t obi_column_get_line_count_from_name(OBIDMS_p dms, 
-                                               const_char_p column_name)
-    
-    
+    int obi_close_header(OBIDMS_column_header_p header)
+
+    int obi_select(OBIDMS_column_p line_selection_column, index_t line_to_grep)
+
+
 cdef extern from "obidmscolumn_int.h" nogil:
+
     int obi_column_set_obiint_with_elt_name(OBIDMS_column_p column, 
-                                            size_t line_nb, 
+                                            index_t line_nb, 
                                             const_char_p element_name, 
                                             obiint_t value)
+
+    int obi_column_set_obiint_with_elt_idx(OBIDMS_column_p column, 
+                                           index_t line_nb, 
+                                           index_t element_idx, 
+                                           obiint_t value)
     
     obiint_t obi_column_get_obiint_with_elt_name(OBIDMS_column_p column, 
-                                                 size_t line_nb, 
+                                                 index_t line_nb, 
                                                  const_char_p element_name)
+    
+    obiint_t obi_column_get_obiint_with_elt_idx(OBIDMS_column_p column, 
+                                                index_t line_nb, 
+                                                index_t element_idx)
 
 cdef extern from "obidmscolumn_bool.h" nogil:
 
     int obi_column_set_obibool_with_elt_name(OBIDMS_column_p column, 
-                                             size_t line_nb, 
+                                             index_t line_nb, 
                                              const_char_p element_name, 
                                              obibool_t value)
+
+    int obi_column_set_obibool_with_elt_idx(OBIDMS_column_p column, 
+                                            index_t line_nb, 
+                                            index_t element_idx, 
+                                            obibool_t value)
     
     obibool_t obi_column_get_obibool_with_elt_name(OBIDMS_column_p column, 
-                                                   size_t line_nb, 
+                                                   index_t line_nb, 
                                                    const_char_p element_name)
+    
+    obibool_t obi_column_get_obibool_with_elt_idx(OBIDMS_column_p column, 
+                                                  index_t line_nb, 
+                                                  index_t element_idx)
 
 cdef extern from "obidmscolumn_char.h" nogil:
 
     int obi_column_set_obichar_with_elt_name(OBIDMS_column_p column, 
-                                             size_t line_nb, 
+                                             index_t line_nb, 
                                              const_char_p element_name, 
                                              obichar_t value)
+
+    int obi_column_set_obichar_with_elt_idx(OBIDMS_column_p column, 
+                                            index_t line_nb, 
+                                            index_t element_idx, 
+                                            obichar_t value)
     
     obichar_t obi_column_get_obichar_with_elt_name(OBIDMS_column_p column, 
-                                                   size_t line_nb, 
+                                                   index_t line_nb, 
                                                    const_char_p element_name)
+    
+    obichar_t obi_column_get_obichar_with_elt_idx(OBIDMS_column_p column, 
+                                                  index_t line_nb, 
+                                                  index_t element_idx)
 
 cdef extern from "obidmscolumn_float.h" nogil:
 
     int obi_column_set_obifloat_with_elt_name(OBIDMS_column_p column, 
-                                              size_t line_nb, 
+                                              index_t line_nb, 
                                               const_char_p element_name, 
                                               obifloat_t value)
+
+    int obi_column_set_obifloat_with_elt_idx(OBIDMS_column_p column, 
+                                             index_t line_nb, 
+                                             index_t element_idx, 
+                                             obifloat_t value)
     
     obifloat_t obi_column_get_obifloat_with_elt_name(OBIDMS_column_p column, 
-                                                     size_t line_nb, 
+                                                     index_t line_nb, 
                                                      const_char_p element_name)
-
-cdef extern from "obidmscolumn_idx.h" nogil:
-
-    int obi_column_set_obiidx_with_elt_name(OBIDMS_column_p column, 
-                                            size_t line_nb, 
-                                            const_char_p element_name, 
-                                            obiidx_t value)
     
-    obiidx_t obi_column_get_obiidx_with_elt_name(OBIDMS_column_p column, 
-                                                 size_t line_nb, 
-                                                 const_char_p element_name)
+    obifloat_t obi_column_get_obifloat_with_elt_idx(OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    index_t element_idx)
+
+cdef extern from "obidmscolumn_str.h" nogil:
+
+    int obi_column_set_obistr_with_elt_name(OBIDMS_column_p column, 
+                                            index_t line_nb, 
+                                            const_char_p element_name, 
+                                            const_char_p value)
+
+    int obi_column_set_obistr_with_elt_idx(OBIDMS_column_p column, 
+                                           index_t line_nb, 
+                                           index_t element_idx, 
+                                           const_char_p value)
+    
+    const_char_p obi_column_get_obistr_with_elt_name(OBIDMS_column_p column, 
+                                                     index_t line_nb, 
+                                                     const_char_p element_name)
+    
+    const_char_p obi_column_get_obistr_with_elt_idx(OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    index_t element_idx)
+
+cdef extern from "obidmscolumn_seq.h" nogil:
+
+    int obi_column_set_obiseq_with_elt_name(OBIDMS_column_p column, 
+                                            index_t line_nb, 
+                                            const_char_p element_name, 
+                                            const_char_p value)
+
+    int obi_column_set_obiseq_with_elt_idx(OBIDMS_column_p column, 
+                                           index_t line_nb, 
+                                           index_t element_idx, 
+                                           const_char_p value)
+    
+    char* obi_column_get_obiseq_with_elt_name(OBIDMS_column_p column, 
+                                              index_t line_nb, 
+                                              const_char_p element_name)
+    
+    char* obi_column_get_obiseq_with_elt_idx(OBIDMS_column_p column, 
+                                             index_t line_nb, 
+                                             index_t element_idx)
 

python/obitools3/obidms/capi/obitaxonomy.pxd

@@ -0,0 +1,42 @@
+#cython: language_level=3
+
+from .obitypes     cimport const_char_p
+from .obidms       cimport OBIDMS_p
+from libc.stdint   cimport int32_t
+
+
+cdef extern from "obidms_taxonomy.h" nogil:
+
+    struct OBIDMS_taxonomy_t
+    ctypedef OBIDMS_taxonomy_t* OBIDMS_taxonomy_p
+    
+    struct ecotxnode :
+        int32_t           taxid
+        int32_t           rank
+        int32_t           farest
+        ecotxnode*        parent
+        char*             name
+    
+    ctypedef ecotxnode ecotx_t
+
+
+    OBIDMS_taxonomy_p obi_read_taxonomy(OBIDMS_p dms, const_char_p taxonomy_name, bint read_alternative_names)
+    
+    int obi_close_taxonomy(OBIDMS_taxonomy_p taxonomy)
+    
+    ecotx_t* obi_taxo_get_parent_at_rank(ecotx_t* taxon, int32_t rankidx)
+    
+    ecotx_t* obi_taxo_get_taxon_with_taxid(OBIDMS_taxonomy_p taxonomy, int32_t taxid)
+    
+    bint obi_taxo_is_taxon_under_taxid(ecotx_t* taxon, int32_t other_taxid)
+    
+    ecotx_t* obi_taxo_get_species(ecotx_t* taxon, OBIDMS_taxonomy_p taxonomy)
+    
+    ecotx_t* obi_taxo_get_genus(ecotx_t* taxon, OBIDMS_taxonomy_p taxonomy)
+    
+    ecotx_t* obi_taxo_get_family(ecotx_t* taxon, OBIDMS_taxonomy_p taxonomy)
+    
+    ecotx_t* obi_taxo_get_kingdom(ecotx_t* taxon, OBIDMS_taxonomy_p taxonomy)
+    
+    ecotx_t* obi_taxo_get_superkingdom(ecotx_t* taxon, OBIDMS_taxonomy_p taxonomy)
+    

python/obitools3/obidms/capi/obitypes.pxd

@@ -1,27 +1,30 @@
 #cython: language_level=3
 
 
-from libc.stdint cimport int32_t
+from libc.stdint cimport int32_t, int64_t
+from posix.types cimport time_t
 
 
 cdef extern from *:
     ctypedef char* const_char_p "const char*"
 
 
-cdef extern from "obidmscolumn.h" nogil:
-
-    ctypedef int32_t obiversion_t
+cdef extern from "encode.h" nogil:
+    bint only_ATGC(const_char_p seq)
 
 
 cdef extern from "obitypes.h" nogil:
 
-    enum OBIType:       # TODO je sais pas si ça sert de declarer le contenu de l'enum
-        OBI_VOID = 0,
+    enum OBIType:
+        OBI_VOID,
         OBI_INT,
         OBI_FLOAT,
         OBI_BOOL,
         OBI_CHAR,
+        OBI_STR,
+        OBI_SEQ,
         OBI_IDX
+
     
     ctypedef OBIType OBIType_t
     
@@ -30,14 +33,18 @@ cdef extern from "obitypes.h" nogil:
     
     ctypedef OBIBool obibool_t
     ctypedef int32_t obiint_t
-    ctypedef double obifloat_t
-    ctypedef char obichar_t  
-    ctypedef size_t obiidx_t
+    ctypedef double  obifloat_t
+    ctypedef char    obichar_t  
+    ctypedef int64_t index_t
 
-    extern obiint_t   OBIInt_NA
-    extern obiidx_t   OBIIdx_NA
-    extern obifloat_t OBIFloat_NA
-    extern obichar_t  OBIChar_NA
-    extern obibool_t  OBIBool_NA
+    ctypedef int32_t obiversion_t
 
-    char* name_data_type(int data_type)
+    extern obiint_t      OBIInt_NA
+    extern index_t       OBIIdx_NA
+    extern obifloat_t    OBIFloat_NA
+    extern obichar_t     OBIChar_NA
+    extern obibool_t     OBIBool_NA
+    extern const_char_p  OBISeq_NA
+    extern const_char_p  OBIStr_NA
+
+    const_char_p name_data_type(int data_type)

python/obitools3/obidms/capi/obiutils.pxd

@@ -0,0 +1,12 @@
+#cython: language_level=3
+
+
+from posix.types cimport time_t
+
+from ..capi.obitypes cimport const_char_p
+
+
+cdef extern from "utils.h" nogil:
+
+    const_char_p obi_format_date(time_t date)
+

python/obitools3/obidms/capi/obiview.pxd

@@ -0,0 +1,249 @@
+#cython: language_level=3
+
+from .obitypes           cimport const_char_p, \
+                                 OBIType_t, \
+                                 obiversion_t, \
+                                 obiint_t, \
+                                 obibool_t, \
+                                 obichar_t, \
+                                 obifloat_t, \
+                                 index_t, \
+                                 time_t
+from ..capi.obidms       cimport OBIDMS_p
+from ..capi.obidmscolumn cimport OBIDMS_column_p
+
+
+cdef extern from "obiview.h" nogil:
+
+    extern const_char_p VIEW_TYPE_NUC_SEQS
+    extern const_char_p NUC_SEQUENCE_COLUMN
+    extern const_char_p ID_COLUMN
+    extern const_char_p DEFINITION_COLUMN
+
+    struct Obiview_t :
+        OBIDMS_p            dms
+        const_char_p        name
+        const_char_p        created_from
+        const_char_p        view_type
+        bint                read_only
+        OBIDMS_column_p     line_selection
+        OBIDMS_column_p     new_line_selection
+        index_t             line_count
+        int                 column_count
+        OBIDMS_column_p     columns
+        const_char_p        comments
+
+    ctypedef Obiview_t* Obiview_p
+
+
+    struct Column_reference_t :
+        const_char_p  column_name
+        obiversion_t  version
+    
+    ctypedef Column_reference_t* Column_reference_p
+
+
+    struct Obiview_infos_t :
+        int                 view_number
+        time_t              creation_date
+        const_char_p        name
+        const_char_p        created_from
+        const_char_p        view_type
+        bint                all_lines
+        Column_reference_t  line_selection
+        index_t             line_count
+        int                 column_count
+        Column_reference_p  column_references
+        const_char_p        comments
+
+    ctypedef Obiview_infos_t* Obiview_infos_p
+
+
+    struct Obiviews_header_t :
+        size_t header_size
+        size_t views_size
+        int    view_count
+    
+    ctypedef Obiviews_header_t* Obiviews_header_p
+
+
+    struct Obiviews_infos_all_t :
+        Obiviews_header_p    header
+        Obiview_infos_p      view_infos
+
+    ctypedef Obiviews_infos_all_t* Obiviews_infos_all_p
+
+
+    Obiview_p obi_new_view_nuc_seqs(OBIDMS_p dms, const_char_p view_name, Obiview_p view_to_clone, index_t* line_selection, const_char_p comments)
+
+    Obiview_p obi_new_view(OBIDMS_p dms, const_char_p view_name, Obiview_p view_to_clone, index_t* line_selection, const_char_p comments)
+    
+    Obiview_p obi_new_view_cloned_from_name(OBIDMS_p dms, const_char_p view_name, const_char_p view_to_clone_name, index_t* line_selection, const_char_p comments)
+
+    Obiview_p obi_new_view_nuc_seqs_cloned_from_name(OBIDMS_p dms, const_char_p view_name, const_char_p view_to_clone_name, index_t* line_selection, const_char_p comments)
+              
+    Obiview_p obi_open_view(OBIDMS_p dms, const_char_p view_name)
+    
+    int obi_view_add_column(Obiview_p view,
+                            const_char_p column_name,
+                            obiversion_t version_number,
+                            OBIType_t data_type,
+                            index_t nb_lines,
+                            index_t nb_elements_per_line,
+                            const_char_p elements_names,
+                            const_char_p indexer_name,
+                            const_char_p comments,
+                            bint create)
+
+    int obi_view_delete_column(Obiview_p view, const_char_p column_name)
+    
+    int obi_select_line(Obiview_p view, index_t line_nb)
+
+    int obi_select_lines(Obiview_p view, index_t* line_nbs)
+        
+    OBIDMS_column_p obi_view_get_column(Obiview_p view, const_char_p column_name)
+
+    OBIDMS_column_p* obi_view_get_pointer_on_column_in_view(Obiview_p view, const_char_p column_name)
+
+    int obi_save_view(Obiview_p view)
+    
+    int obi_close_view(Obiview_p view)
+    
+    int obi_save_and_close_view(Obiview_p view)
+    
+    Obiviews_infos_all_p obi_read_view_infos(OBIDMS_p dms)
+
+    int obi_close_view_infos(Obiviews_infos_all_p views)
+
+    int obi_column_set_obiint_with_elt_name_in_view(Obiview_p view,
+                                                    OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    const_char_p element_name, 
+                                                    obiint_t value)
+
+    int obi_column_set_obiint_with_elt_idx_in_view(Obiview_p view,
+                                                   OBIDMS_column_p column, 
+                                                   index_t line_nb, 
+                                                   index_t element_idx, 
+                                                   obiint_t value)
+    
+    obiint_t obi_column_get_obiint_with_elt_name_in_view(Obiview_p view,
+                                                         OBIDMS_column_p column, 
+                                                         index_t line_nb, 
+                                                         const_char_p element_name)
+    
+    obiint_t obi_column_get_obiint_with_elt_idx_in_view(Obiview_p view,
+                                                        OBIDMS_column_p column, 
+                                                        index_t line_nb, 
+                                                        index_t element_idx)
+
+    int obi_column_set_obibool_with_elt_name_in_view(Obiview_p view,
+                                                     OBIDMS_column_p column, 
+                                                     index_t line_nb, 
+                                                     const_char_p element_name, 
+                                                     obibool_t value)
+
+    int obi_column_set_obibool_with_elt_idx_in_view(Obiview_p view,
+                                                    OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    index_t element_idx, 
+                                                    obibool_t value)
+    
+    obibool_t obi_column_get_obibool_with_elt_name_in_view(Obiview_p view,
+                                                           OBIDMS_column_p column, 
+                                                           index_t line_nb, 
+                                                           const_char_p element_name)
+    
+    obibool_t obi_column_get_obibool_with_elt_idx_in_view(Obiview_p view,
+                                                          OBIDMS_column_p column, 
+                                                          index_t line_nb, 
+                                                          index_t element_idx)
+
+    int obi_column_set_obichar_with_elt_name_in_view(Obiview_p view,
+                                                     OBIDMS_column_p column, 
+                                                     index_t line_nb, 
+                                                     const_char_p element_name, 
+                                                     obichar_t value)
+
+    int obi_column_set_obichar_with_elt_idx_in_view(Obiview_p view,
+                                                    OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    index_t element_idx, 
+                                                    obichar_t value)
+    
+    obichar_t obi_column_get_obichar_with_elt_name_in_view(Obiview_p view,
+                                                           OBIDMS_column_p column, 
+                                                           index_t line_nb, 
+                                                           const_char_p element_name)
+    
+    obichar_t obi_column_get_obichar_with_elt_idx_in_view(Obiview_p view,
+                                                          OBIDMS_column_p column, 
+                                                          index_t line_nb, 
+                                                          index_t element_idx)
+
+    int obi_column_set_obifloat_with_elt_name_in_view(Obiview_p view,
+                                                      OBIDMS_column_p column, 
+                                                      index_t line_nb, 
+                                                      const_char_p element_name, 
+                                                      obifloat_t value)
+
+    int obi_column_set_obifloat_with_elt_idx_in_view(Obiview_p view,
+                                                     OBIDMS_column_p column, 
+                                                     index_t line_nb, 
+                                                     index_t element_idx, 
+                                                     obifloat_t value)
+    
+    obifloat_t obi_column_get_obifloat_with_elt_name_in_view(Obiview_p view,
+                                                             OBIDMS_column_p column, 
+                                                             index_t line_nb, 
+                                                             const_char_p element_name)
+    
+    obifloat_t obi_column_get_obifloat_with_elt_idx_in_view(Obiview_p view,
+                                                            OBIDMS_column_p column, 
+                                                            index_t line_nb, 
+                                                            index_t element_idx)
+
+    int obi_column_set_obistr_with_elt_name_in_view(Obiview_p view,
+                                                    OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    const_char_p element_name, 
+                                                    const_char_p value)
+
+    int obi_column_set_obistr_with_elt_idx_in_view(Obiview_p view,
+                                                   OBIDMS_column_p column, 
+                                                   index_t line_nb, 
+                                                   index_t element_idx, 
+                                                   const_char_p value)
+    
+    const_char_p obi_column_get_obistr_with_elt_name_in_view(Obiview_p view,
+                                                           OBIDMS_column_p column, 
+                                                           index_t line_nb, 
+                                                           const_char_p element_name)
+    
+    const_char_p obi_column_get_obistr_with_elt_idx_in_view(Obiview_p view,
+                                                            OBIDMS_column_p column, 
+                                                            index_t line_nb, 
+                                                            index_t element_idx)
+
+    int obi_column_set_obiseq_with_elt_name_in_view(Obiview_p view,
+                                                    OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    const_char_p element_name, 
+                                                    const_char_p value)
+
+    int obi_column_set_obiseq_with_elt_idx_in_view(Obiview_p view,
+                                                   OBIDMS_column_p column, 
+                                                   index_t line_nb, 
+                                                   index_t element_idx, 
+                                                   const_char_p value)
+    
+    char* obi_column_get_obiseq_with_elt_name_in_view(Obiview_p view,
+                                                      OBIDMS_column_p column, 
+                                                      index_t line_nb, 
+                                                      const_char_p element_name)
+    
+    char* obi_column_get_obiseq_with_elt_idx_in_view(Obiview_p view,
+                                                     OBIDMS_column_p column, 
+                                                     index_t line_nb, 
+                                                     index_t element_idx)
+

python/obitools3/obigrep.py

@@ -0,0 +1,51 @@
+import sys
+import argparse
+
+from obitools3.obidms._obidms import OBIDMS
+
+if __name__ == '__main__': 
+    
+    parser = argparse.ArgumentParser(description='Pseudo obigrep.')
+    
+    parser.add_argument('-V', '--view', dest='view', type=str,
+                       help='Name of the view that should be considered')    
+
+    parser.add_argument('-N', '--new_view', dest='new_view', type=str,
+                       help='Name of the new view that should be created')    
+    
+#     parser.add_argument('-k', '--key', dest='key', type=str,
+#                        help='Name of the key that should be considered')
+# 
+#     parser.add_argument('-c', '--comp', dest='comparison', type=int,
+#                        help='Comparison to be made: -1:< ; 0:== ; 1:>')
+# 
+#     parser.add_argument('-v', '--value', dest='value', type=object,
+#                        help='Value to be compared')
+
+    args = parser.parse_args()
+    
+    d = OBIDMS('tdms')
+    
+    #condition = 1
+    line_selec = []
+    
+    v = d.open_view(args.view)
+    
+    i = 0
+    for l in v :
+        if l['score'] > 350 : 
+            line_selec.append(i)
+        i+=1
+        
+    new_v = d.new_view(args.new_view, view_to_clone=v, line_selection=line_selec, view_type="NUC_SEQS_VIEW", comments="obigrep "+args.view+" to "+args.new_view) #args.key+" "+str(args.comparison)+" "+str(args.value)+" "+)
+
+    print("\n")
+    print(new_v.__repr__())
+
+    v.save_and_close()
+    new_v.save_and_close()
+    
+    d.close()
+
+    print("\nDone.")
+    

python/obitools3/obihead.py

@@ -0,0 +1,43 @@
+import sys
+import argparse
+
+from obitools3.obidms._obidms import OBIDMS
+
+if __name__ == '__main__': 
+    
+    parser = argparse.ArgumentParser(description='Pseudo obihead.')
+    
+    parser.add_argument('-V', '--view', dest='view', type=str,
+                       help='Name of the view that should be considered')    
+
+    parser.add_argument('-N', '--new_view', dest='new_view', type=str,
+                       help='Name of the new view that should be created')    
+    
+    parser.add_argument('-n', '--nb', dest='nb_lines', type=int,
+                       help='Number of lines that should be taken')    
+    
+
+    args = parser.parse_args()
+    
+    d = OBIDMS('tdms')
+    
+    #condition = 1
+    line_selec = []
+    
+    v = d.open_view(args.view)
+    
+    for i in range(0, args.nb_lines) :
+        line_selec.append(i)
+        
+    new_v = d.new_view(args.new_view, view_to_clone=v, line_selection=line_selec, view_type="NUC_SEQS_VIEW", comments="obihead "+str(args.nb_lines)+", "+args.view+" to "+args.new_view) #args.key+" "+str(args.comparison)+" "+str(args.value)+" "+)
+
+    print("\n")
+    print(new_v.__repr__())
+
+    v.save_and_close()
+    new_v.save_and_close()
+    
+    d.close()
+
+    print("\nDone.")
+    

python/obitools3/obiimport.py

@@ -0,0 +1,199 @@
+import sys
+import argparse
+import time
+
+from obitools3.obidms._obidms import OBIDMS
+
+
+def bufferedRead(fileobj,size=209715200):   ## 200 MB
+   buffer = fileobj.readlines(size)
+   while buffer:
+       for l in buffer:
+           yield l
+       buffer = fileobj.readlines(size)
+
+
+if __name__ == '__main__': 
+    
+    parser = argparse.ArgumentParser(description='Convert a fasta file in an OBIDMS.')
+    
+    parser.add_argument('-i', '--input', dest='input_file', type=str,
+                       help='Name of the file containing the sequences')
+
+    args = parser.parse_args()
+
+    d = OBIDMS('tdms')
+    
+    view = d.new_view('uniq view', view_type="NUC_SEQS_VIEW")
+
+#    for i in range(35000000) :
+#        if (not (i%500000)) :
+#            print(str(time.time())+'\t'+str(i))
+#        id = "@HWI-D00405:142:C71BAANXX:4:1101:1234:2234_CONS_SUB_SUB_"+str(i)
+#        view[i].set_id(id)
+#        if id != view[i]["ID"] :
+#            print("nope", id, view[i]["ID"])
+    
+    input_file = open(args.input_file, 'r')
+    input_file_buffered = bufferedRead(input_file)
+    
+#         
+#     if args.input_file[-1:] == "a" :
+#     
+#         i = 0
+#         next = False
+#         first = True
+#         
+#         for line in input_file :
+#             
+#             if line[0] == ">" :
+#                 
+#                 if not first :
+#                     # save seq
+#                     #print(i, id, seq)
+#                     view[i].set_sequence(seq)
+#                     i+=1
+#                 
+#                 first = False
+#                     
+#                 #id = line.split(" ", 1)[0][1:]
+#                 #rest = (line[:-1].split(" ", 1)[1]).split(";")
+#                 #view[i].set_id(id)
+#                 
+# #                 description = ""
+# #                 for j in range(len(rest)) :
+# #                     if "=" in rest[j] :
+# #                         rest[j] = rest[j].strip()
+# #                         rest[j] = rest[j].split("=", 1)
+# #                         column_name = rest[j][0]
+# #                         v = rest[j][1]
+# #                         if ((not v.isalpha()) and (v.isalnum())) :
+# #                             conv_v = int(v)
+# #                         elif (v == "True") or (v == "False") :
+# #                             conv_v = bool(v)
+# #                         else :
+# #                             f = True
+# #                             for letter in v :
+# #                                 if ((not letter.isalnum()) or (letter != ".")) :
+# #                                     f = False
+# #                             if f :
+# #                                 conv_v = float(v)
+# #                             else : 
+# #                                 conv_v = v
+# #                         view[i][column_name] = conv_v
+# #                     else :
+# #                         description+=rest[j]
+# #                 
+# #                 if description != "" :
+# #                     description = description.strip()
+# #                     view[i].set_description(description)
+#                     
+#                 #print(id)
+#                 #print(rest)
+#                 #print(description)
+#                 
+#                 next = True
+#                 
+#             elif next == True :
+#                 
+#      #           if not (i % 1E5) :
+#      #               print(i)
+#                 
+#                 seq = line[:-1]
+#                 next = False
+#             
+#             elif not next :
+#                 
+#                 seq += line[:-1]
+# 
+# 
+#     elif args.input_file[-1:] == "q" :
+#         
+#         i = 0
+#         l = 0
+#         next = False
+#
+    l=0
+    i=0
+#     while (True):
+#         l+=1
+#         line = input_file.readline()
+#         if line=="":
+#             break
+    for line in input_file_buffered :
+# 
+             #if i > 1E7 :
+#             #    print('hmm?')
+#                         
+#        if i == 6000000 :
+#            break
+#             
+        if l%4 == 0 :
+#                
+            if (not (i%500000)) :
+                print(str(time.time())+'\t'+str(i))
+# # 
+# #                 #print("header", line)
+# # 
+            id = line.split(" ", 1)[0][1:]
+#            print(id)
+# #                 #rest = (line[:-1].split(" ", 1)[1]).split(";")
+            view[i].set_id(id)
+#            print(view[i]["ID"])
+# 
+#            i+=1
+        
+#        l+=1
+#                 
+# #                 description = ""
+# #                 for j in range(len(rest)) :
+# #                     if "=" in rest[j] :
+# #                         rest[j] = rest[j].strip()
+# #                         rest[j] = rest[j].split("=", 1)
+# #                         column_name = rest[j][0]
+# #                         #print("COLUMN", column_name)
+# #                         v = rest[j][1]
+# #                         if (v == "") and (column_name in view) and (view[column_name].get_data_type() == "OBI_SEQ") :
+# #                             #print(">>>>>>YUP")
+# #                             conv_v = "aa"
+# #                         else :
+# #                             if ((not v.isalpha()) and (v.isalnum())) :
+# #                                 conv_v = int(v)
+# #                             elif (v == "True") or (v == "False") :
+# #                                 conv_v = bool(v)
+# #                             else :
+# #                                 f = True
+# #                                 for letter in v :
+# #                                     if ((not letter.isalnum()) or (letter != ".")) :
+# #                                         f = False
+# #                                 if f :
+# #                                     conv_v = float(v)
+# #                                 else :
+# #                                     conv_v = v
+# #                         view[i][column_name] = conv_v
+# #                     else :
+# #                         description+=rest[j]
+# #                 
+# #                 if description != "" :
+# #                     description = description.strip()
+# #                     view[i].set_description(description)
+#                 
+        elif l%4 == 1 :
+# #                
+            seq = line[:-1]
+#                 #print("seq", seq)
+            view[i].set_sequence(seq)
+            i+=1
+#                         
+        l+=1
+#     
+#     
+    input_file.close()
+
+    #print(view)
+    print(view.__repr__())
+
+    view.save_and_close()
+    d.close()
+
+    print("Done.")

python/obitools3/parsers/fasta.pxd

@@ -0,0 +1,8 @@
+#cython: language_level=3
+
+from .header cimport parseHeader
+from ..files.universalopener cimport uopen
+from ..files.linebuffer cimport LineBuffer
+
+
+    

python/obitools3/parsers/fasta.pyx

@@ -0,0 +1,46 @@
+#cython: language_level=3
+
+'''
+Created on 30 mars 2016
+
+@author: coissac
+'''
+
+def fastaIterator(lineiterator, int buffersize=100000000):
+    cdef LineBuffer lb
+    cdef str        ident
+    cdef str        definition
+    cdef dict       tags
+    cdef list       s
+    
+    if isinstance(lineiterator,(str,bytes)):
+        lineiterator=uopen(lineiterator)
+    
+    if isinstance(lineiterator, LineBuffer):
+        lb=lineiterator
+    else:
+        lb=LineBuffer(lineiterator,buffersize)
+        
+    i = iter(lb)
+    line = next(i)
+    
+    while True:
+        ident,tags,definition = parseHeader(line)
+        s = []
+        line = next(i)
+        while line[0]!='>':
+            s.append(line[0:-1])
+            line = next(i)
+        sequence  = "".join(s)
+        quality   = None
+
+        yield { "id"         : ident,
+                "definition" : definition,
+                "sequence"   : sequence,
+                "quality"    : quality,
+                "tags"       : tags,
+                "annotation" : {}
+              }
+        
+        
+        

python/obitools3/parsers/fastq.pxd

@@ -0,0 +1,8 @@
+#cython: language_level=3
+
+from .header cimport parseHeader
+from ..files.universalopener cimport uopen
+from ..files.linebuffer cimport LineBuffer
+
+
+    

python/obitools3/parsers/fastq.pyx

@@ -0,0 +1,41 @@
+#cython: language_level=3
+
+'''
+Created on 30 mars 2016
+
+@author: coissac
+'''
+
+
+
+def fastqIterator(lineiterator, int buffersize=100000000):
+    cdef LineBuffer lb
+    cdef str        ident
+    cdef str        definition
+    cdef dict       tags
+    
+    if isinstance(lineiterator,(str,bytes)):
+        lineiterator=uopen(lineiterator)
+    
+    if isinstance(lineiterator, LineBuffer):
+        lb=lineiterator
+    else:
+        lb=LineBuffer(lineiterator,buffersize)
+        
+    i = iter(lb)
+    for line in i:
+        ident,tags,definition = parseHeader(line)
+        sequence  = next(i)[0:-1]
+        next(i)
+        quality   = next(i)[0:-1]
+
+        yield { "id"         : ident,
+                "definition" : definition,
+                "sequence"   : sequence,
+                "quality"    : quality,
+                "tags"       : tags,
+                "annotation" : {}
+              }
+        
+        
+        

python/obitools3/parsers/header.pxd

@@ -0,0 +1,5 @@
+#cython: language_level=3
+
+cdef object __etag__(str x)
+
+cpdef tuple parseHeader(str header)

python/obitools3/parsers/header.pyx

@@ -0,0 +1,78 @@
+#cython: language_level=3
+
+'''
+Created on 25 mars 2016
+
+@author: coissac
+'''
+
+import re
+
+__ret__         = re.compile('''(([^ ]+)=('[^']*'|"[^"]*"|[^;]+); *)+?''')
+__re_int__      = re.compile("^[+-]?[0-9]+$")
+__re_float__    = re.compile("^[+-]?[0-9]+(\.[0-9]*)?([eE][+-]?[0-9]+)?$")
+__re_str__      = re.compile("""^"[^"]*"|'[^']*'$""")
+__re_dict__     = re.compile("""^\{\ *
+                                   (
+                                       ("[^"]*"|'[^']*')
+                                        \ *:\ *
+                                       ([^,}]+|
+                                        "[^"]*"|
+                                        '[^']*'
+                                       )
+                                   )?
+                                   (\ *,\ *
+                                       ("[^"]*"|'[^']*')
+                                        \ *:\ *
+                                       ([^,}]+|
+                                        "[^"]*"|
+                                        '[^']*'
+                                       )
+                                    )*\ *\}$""", re.VERBOSE)
+
+
+cdef object __etag__(str x):
+    if __re_int__.match(x):
+        v=int(x)
+    elif __re_float__.match(x):
+        v=float(x)
+    elif __re_str__.match(x):
+        v=x[1:-1]
+    elif x=='False':
+        v=False
+    elif x=='True':
+        v=True
+    elif __re_dict__.match(x):
+        v=eval(x)
+    else:
+        v=x
+    return v
+
+cpdef tuple parseHeader(str header):
+    cdef list m 
+    cdef dict tags 
+    cdef str definition
+    cdef str ident
+    cdef str second
+    
+    m=header[1:-1].split(maxsplit=1)
+
+    ident=m[0]
+    
+    if len(m)==1:
+        tags={}
+        definition='' 
+    else:
+        second=m[1]
+        m = __ret__.findall(second)
+        
+        if m:
+            tags = dict([(a[1],__etag__(a[2])) for a in m])
+            definition = second.split(m[-1][0],1)[1].strip()
+        else:
+            tags = {}
+            definition = second.strip() 
+        
+    return ident,tags,definition
+    
+    

python/obitools3/unit_tests.py

@@ -0,0 +1,311 @@
+import os
+import sys
+import shutil
+import unittest
+from random import randint, uniform, choice
+import string
+
+from obitools3.obidms._obidms import OBIDMS
+
+
+LINE_COUNT_FOR_TEST_COLUMN = 10000         # TODO randomize?
+SMALLER_LINE_COUNT_FOR_TEST_COLUMN = 1000  # TODO randomize?
+NB_ELEMENTS_PER_LINE = 10                  # TODO randomize?
+DMS_NAME = "unit_test_dms"
+
+
+def create_test_obidms():
+    dms_name = DMS_NAME
+    dms_dir_name = dms_name+'.obidms'
+    dms = OBIDMS(dms_name)
+    return (dms, dms_name, dms_dir_name)
+
+
+def create_test_column(dms, data_type, multiple_elements_per_line=False):
+    
+    col_name = "unit_test_"+data_type
+
+    if multiple_elements_per_line :
+        elts_names = elements_names()
+        col = dms.open_column(col_name, 
+                              create=True, 
+                              type=data_type, 
+                              nb_elements_per_line=NB_ELEMENTS_PER_LINE,
+                              elements_names=elts_names)
+        return (col, col_name, elts_names)
+
+    else :
+        col = dms.open_column(col_name, 
+                              create=True, 
+                              type=data_type)
+        return (col, col_name)
+
+
+def elements_names():
+    names = [str(i) for i in range(NB_ELEMENTS_PER_LINE)]
+    return names
+
+
+def random_obivalue(data_type):
+    r = 1000000
+    if data_type == "OBI_INT" :
+        return randint(-r,r)
+    elif data_type == "OBI_FLOAT" :
+        return uniform(-r,r)
+    elif data_type == "OBI_BOOL" :
+        return randint(0,1)
+    elif data_type == "OBI_CHAR" :
+        return choice(string.ascii_lowercase)
+    elif data_type == "OBI_STR" :
+        length = randint(1,200)
+        randoms = ''.join(choice(string.ascii_lowercase) for i in range(length))
+        return randoms
+    elif data_type == "OBI_SEQ" :
+        length = randint(1,200)
+        randoms = ''.join(choice("atgcryswkmdbhvn") for i in range(length))
+        return randoms
+
+class OBIDMS_Column_TestCase(unittest.TestCase):
+    def tearDown(self):
+        self.col.close()
+        self.dms.close()
+        shutil.rmtree(self.dms_dir_name, ignore_errors=True)
+    def test_OBIDMS_column_type(self):
+        assert self.col.get_data_type() == self.data_type, 'Wrong data type associated with column'
+    def test_OBIDMS_column_cloning(self):
+        for i in range(LINE_COUNT_FOR_TEST_COLUMN):
+            self.col[i]= random_obivalue(self.data_type)
+        self.col.close()
+        clone = self.dms.open_column(self.col_name, clone=True)
+        self.col = self.dms.open_column(self.col_name)
+        assert clone.get_nb_lines_used() == self.col.get_nb_lines_used(), "Cloned column doesn't have the same number of lines used"
+        i=0
+        for i in range(clone.get_nb_lines_used()) :
+            assert clone[i] == self.col[i], "Different value in original column and cloned column"
+            assert clone[i] is not None, "None value"
+        clone.close()
+    def test_OBIDMS_column_set_and_get(self):
+        for i in range(LINE_COUNT_FOR_TEST_COLUMN):
+            v = random_obivalue(self.data_type)
+            self.col[i] = v
+            assert self.col[i] == v, "Different value than the set value"
+            assert self.col[i] is not None, "None value"
+    def test_OBIDMS_referring_column(self):
+        for i in range(LINE_COUNT_FOR_TEST_COLUMN):
+            self.col[i] = random_obivalue(self.data_type)
+        ref_col = self.dms.open_column(self.col_name, referring=True)
+        j = 0
+        for i in range(LINE_COUNT_FOR_TEST_COLUMN):
+            if i%2 :    # TODO randomize
+                ref_col.grep_line(i)
+                assert ref_col[j] == self.col[i], "Different value in original column and returned by referring column"
+                assert ref_col[j] is not None, "None value"
+                j+=1
+
+
+class OBIDMS_Column_multiple_elements_TestCase(OBIDMS_Column_TestCase):
+    def test_OBIDMS_column_cloning(self):
+        pass
+        for i in range(SMALLER_LINE_COUNT_FOR_TEST_COLUMN):
+            v = {}
+            for e in self.elts_names :
+                v[e] = random_obivalue(self.data_type)
+            self.col[i] = v
+        self.col.close()
+        clone = self.dms.open_column(self.col_name, clone=True)
+        self.col = self.dms.open_column(self.col_name)
+        assert clone.get_nb_lines_used() == self.col.get_nb_lines_used(), "Cloned column doesn't have the same number of lines used"
+        i=0
+        for i in range(SMALLER_LINE_COUNT_FOR_TEST_COLUMN):
+            assert self.col[i] == clone[i], "Different value in original column and cloned column"
+            assert self.col[i] is not None, "None value"
+        clone.close()
+    def test_OBIDMS_column_set_and_get_with_elements_names(self):
+        for i in range(SMALLER_LINE_COUNT_FOR_TEST_COLUMN):
+            for e in range(NB_ELEMENTS_PER_LINE) :
+                v = random_obivalue(self.data_type)
+                self.col.set_item(i, self.elts_names[e], v)
+                assert self.col.get_item(i, self.elts_names[e]) == v, "Different value than the set value"
+                assert self.col.get_item(i, self.elts_names[e]) is not None, "None value"
+    def test_OBIDMS_column_set_and_get(self):
+        for i in range(SMALLER_LINE_COUNT_FOR_TEST_COLUMN):
+            v = {}
+            for e in self.elts_names :
+                v[e] = random_obivalue(self.data_type)
+            self.col[i] = v
+            assert self.col[i] == v, "Different value than the set value"
+            assert self.col[i] is not None, "None value"
+    def test_OBIDMS_referring_column(self):
+        for i in range(SMALLER_LINE_COUNT_FOR_TEST_COLUMN):
+            v = {}
+            for e in self.elts_names :
+                v[e] = random_obivalue(self.data_type)
+            self.col[i] = v
+        ref_col = self.dms.open_column(self.col_name, referring=True)
+        j = 0
+        for i in range(SMALLER_LINE_COUNT_FOR_TEST_COLUMN):
+            if i%2 :    # TODO randomize
+                ref_col.grep_line(i)
+                assert ref_col[j] == self.col[i], "Different value in original column and returned by referring column"
+                assert ref_col[j] is not None, "None value"
+                j+=1
+        ref_col.close()
+
+
+class OBIDMS_Column_OBI_INT_TestCase(OBIDMS_Column_TestCase):
+    def setUp(self):
+        self.data_type = 'OBI_INT'
+        self.dms, \
+        self.dms_name, \
+        self.dms_dir_name = create_test_obidms()
+        self.col, \
+        self.col_name = create_test_column(self.dms, 
+                                           self.data_type)
+       
+
+class OBIDMS_Column_OBI_INT_multiple_elements_TestCase(OBIDMS_Column_multiple_elements_TestCase):
+    def setUp(self):
+        self.data_type = 'OBI_INT'
+        self.dms, \
+        self.dms_name, \
+        self.dms_dir_name = create_test_obidms()
+        self.col, \
+        self.col_name, \
+        self.elts_names = create_test_column(self.dms, 
+                                             self.data_type,
+                                             multiple_elements_per_line=True)
+
+
+class OBIDMS_Column_OBI_FLOAT_TestCase(OBIDMS_Column_TestCase):
+    def setUp(self):
+        self.data_type = 'OBI_FLOAT'
+        self.dms, \
+        self.dms_name, \
+        self.dms_dir_name = create_test_obidms()
+        self.col, \
+        self.col_name = create_test_column(self.dms, 
+                                           self.data_type)
+       
+
+class OBIDMS_Column_OBI_FLOAT_multiple_elements_TestCase(OBIDMS_Column_multiple_elements_TestCase):
+    def setUp(self):
+        self.data_type = 'OBI_FLOAT'
+        self.dms, \
+        self.dms_name, \
+        self.dms_dir_name = create_test_obidms()
+        self.col, \
+        self.col_name, \
+        self.elts_names = create_test_column(self.dms, 
+                                             self.data_type,
+                                             multiple_elements_per_line=True)
+
+
+class OBIDMS_Column_OBI_BOOL_TestCase(OBIDMS_Column_TestCase):
+    def setUp(self):
+        self.data_type = 'OBI_BOOL'
+        self.dms, \
+        self.dms_name, \
+        self.dms_dir_name = create_test_obidms()
+        self.col, \
+        self.col_name = create_test_column(self.dms, 
+                                           self.data_type)
+       
+
+class OBIDMS_Column_OBI_BOOL_multiple_elements_TestCase(OBIDMS_Column_multiple_elements_TestCase):
+    def setUp(self):
+        self.data_type = 'OBI_BOOL'
+        self.dms, \
+        self.dms_name, \
+        self.dms_dir_name = create_test_obidms()
+        self.col, \
+        self.col_name, \
+        self.elts_names = create_test_column(self.dms, 
+                                             self.data_type,
+                                             multiple_elements_per_line=True)
+
+
+class OBIDMS_Column_OBI_CHAR_TestCase(OBIDMS_Column_TestCase):
+    def setUp(self):
+        self.data_type = 'OBI_CHAR'
+        self.dms, \
+        self.dms_name, \
+        self.dms_dir_name = create_test_obidms()
+        self.col, \
+        self.col_name = create_test_column(self.dms, 
+                                                self.data_type)
+       
+
+class OBIDMS_Column_OBI_CHAR_multiple_elements_TestCase(OBIDMS_Column_multiple_elements_TestCase):
+    def setUp(self):
+        self.data_type = 'OBI_CHAR'
+        self.dms, \
+        self.dms_name, \
+        self.dms_dir_name = create_test_obidms()
+        self.col, \
+        self.col_name, \
+        self.elts_names = create_test_column(self.dms, 
+                                             self.data_type,
+                                             multiple_elements_per_line=True)
+
+
+class OBIDMS_Column_OBI_STR_TestCase(OBIDMS_Column_TestCase):
+    def setUp(self):
+        self.data_type = 'OBI_STR'
+        self.dms, \
+        self.dms_name, \
+        self.dms_dir_name = create_test_obidms()
+        self.col, \
+        self.col_name = create_test_column(self.dms, 
+                                           self.data_type)
+       
+
+class OBIDMS_Column_OBI_STR_multiple_elements_TestCase(OBIDMS_Column_multiple_elements_TestCase):
+    def setUp(self):
+        self.data_type = 'OBI_STR'
+        self.dms, \
+        self.dms_name, \
+        self.dms_dir_name = create_test_obidms()
+        self.col, \
+        self.col_name, \
+        self.elts_names = create_test_column(self.dms, 
+                                             self.data_type,
+                                             multiple_elements_per_line=True)
+
+class OBIDMS_Column_OBI_SEQ_TestCase(OBIDMS_Column_TestCase):
+    def setUp(self):
+        self.data_type = 'OBI_SEQ'
+        self.dms, \
+        self.dms_name, \
+        self.dms_dir_name = create_test_obidms()
+        self.col, \
+        self.col_name = create_test_column(self.dms, 
+                                           self.data_type)
+       
+
+class OBIDMS_Column_OBI_SEQ_multiple_elements_TestCase(OBIDMS_Column_multiple_elements_TestCase):
+    def setUp(self):
+        self.data_type = 'OBI_SEQ'
+        self.dms, \
+        self.dms_name, \
+        self.dms_dir_name = create_test_obidms()
+        self.col, \
+        self.col_name, \
+        self.elts_names = create_test_column(self.dms, 
+                                             self.data_type,
+                                             multiple_elements_per_line=True)
+
+if __name__ == '__main__': 
+     unittest.main(verbosity=2, defaultTest=["OBIDMS_Column_OBI_INT_TestCase",
+                                             "OBIDMS_Column_OBI_INT_multiple_elements_TestCase",
+                                             "OBIDMS_Column_OBI_FLOAT_TestCase",
+                                             "OBIDMS_Column_OBI_FLOAT_multiple_elements_TestCase",
+                                             "OBIDMS_Column_OBI_BOOL_TestCase",
+                                             "OBIDMS_Column_OBI_BOOL_multiple_elements_TestCase",
+                                             "OBIDMS_Column_OBI_CHAR_TestCase",
+                                             "OBIDMS_Column_OBI_CHAR_multiple_elements_TestCase",
+                                             "OBIDMS_Column_OBI_STR_TestCase",
+                                             "OBIDMS_Column_OBI_STR_multiple_elements_TestCase",
+                                             "OBIDMS_Column_OBI_SEQ_TestCase",
+                                             "OBIDMS_Column_OBI_SEQ_multiple_elements_TestCase"])
+
+

python/obitools3/utils.pxd

@@ -1,5 +1,6 @@
 #cython: language_level=3
 
+
 cdef bytes str2bytes(str string)
 
 cdef str bytes2str(bytes string)

python/obitools3/utils.pyx

@@ -1,7 +1,28 @@
 #cython: language_level=3
 
+import sys
+import io
+
 cdef bytes str2bytes(str string):
+    """
+    Short cut to convert ascii encoded python string (str) to bytes 
+    which can be easily converted to C-strings. 
+    
+        @param string: the python string to be converted.
+        @type string: str
+        @return a transcoded string
+        @rtype: bytes 
+    """
     return string.encode('ascii')
 
 cdef str bytes2str(bytes string):
-    return string.decode('ascii')
+    """
+    Short cut to convert bytes (C-strings) to ascii encoded python string (str).
+    
+        @param string: the binary (C-string) string to be converted.
+        @type string: bytes
+        @return an ascii transcoded string
+        @rtype: str 
+    """
+    return string.decode('ascii')
+

python/obitools3/version.py

@@ -1,5 +1,5 @@
-major = 1
-minor = 1
-serial= '16'
+major = 0
+minor = 0
+serial= '0'
 
-version ="%2d.%02d.%s" % (major,minor,serial)
+version ="%d.%02d.%s" % (major,minor,serial)