Abandoned beginning of functions to apply comparisons to whole columns

(not worth it for now as comparisons are already fast enough)

LICENSE

@@ -0,0 +1,518 @@
+
+  CeCILL FREE SOFTWARE LICENSE AGREEMENT
+
+Version 2.1 dated 2013-06-21
+
+
+    Notice
+
+This Agreement is a Free Software license agreement that is the result
+of discussions between its authors in order to ensure compliance with
+the two main principles guiding its drafting:
+
+  * firstly, compliance with the principles governing the distribution
+    of Free Software: access to source code, broad rights granted to users,
+  * secondly, the election of a governing law, French law, with which it
+    is conformant, both as regards the law of torts and intellectual
+    property law, and the protection that it offers to both authors and
+    holders of the economic rights over software.
+
+The authors of the CeCILL (for Ce[a] C[nrs] I[nria] L[ogiciel] L[ibre]) 
+license are: 
+
+Commissariat à l'énergie atomique et aux énergies alternatives - CEA, a
+public scientific, technical and industrial research establishment,
+having its principal place of business at 25 rue Leblanc, immeuble Le
+Ponant D, 75015 Paris, France.
+
+Centre National de la Recherche Scientifique - CNRS, a public scientific
+and technological establishment, having its principal place of business
+at 3 rue Michel-Ange, 75794 Paris cedex 16, France.
+
+Institut National de Recherche en Informatique et en Automatique -
+Inria, a public scientific and technological establishment, having its
+principal place of business at Domaine de Voluceau, Rocquencourt, BP
+105, 78153 Le Chesnay cedex, France.
+
+
+    Preamble
+
+The purpose of this Free Software license agreement is to grant users
+the right to modify and redistribute the software governed by this
+license within the framework of an open source distribution model.
+
+The exercising of this right is conditional upon certain obligations for
+users so as to preserve this status for all subsequent redistributions.
+
+In consideration of access to the source code and the rights to copy,
+modify and redistribute granted by the license, users are provided only
+with a limited warranty and the software's author, the holder of the
+economic rights, and the successive licensors only have limited liability.
+
+In this respect, the risks associated with loading, using, modifying
+and/or developing or reproducing the software by the user are brought to
+the user's attention, given its Free Software status, which may make it
+complicated to use, with the result that its use is reserved for
+developers and experienced professionals having in-depth computer
+knowledge. Users are therefore encouraged to load and test the
+suitability of the software as regards their requirements in conditions
+enabling the security of their systems and/or data to be ensured and,
+more generally, to use and operate it in the same conditions of
+security. This Agreement may be freely reproduced and published,
+provided it is not altered, and that no provisions are either added or
+removed herefrom.
+
+This Agreement may apply to any or all software for which the holder of
+the economic rights decides to submit the use thereof to its provisions.
+
+Frequently asked questions can be found on the official website of the
+CeCILL licenses family (http://www.cecill.info/index.en.html) for any 
+necessary clarification.
+
+
+    Article 1 - DEFINITIONS
+
+For the purpose of this Agreement, when the following expressions
+commence with a capital letter, they shall have the following meaning:
+
+Agreement: means this license agreement, and its possible subsequent
+versions and annexes.
+
+Software: means the software in its Object Code and/or Source Code form
+and, where applicable, its documentation, "as is" when the Licensee
+accepts the Agreement.
+
+Initial Software: means the Software in its Source Code and possibly its
+Object Code form and, where applicable, its documentation, "as is" when
+it is first distributed under the terms and conditions of the Agreement.
+
+Modified Software: means the Software modified by at least one
+Contribution.
+
+Source Code: means all the Software's instructions and program lines to
+which access is required so as to modify the Software.
+
+Object Code: means the binary files originating from the compilation of
+the Source Code.
+
+Holder: means the holder(s) of the economic rights over the Initial
+Software.
+
+Licensee: means the Software user(s) having accepted the Agreement.
+
+Contributor: means a Licensee having made at least one Contribution.
+
+Licensor: means the Holder, or any other individual or legal entity, who
+distributes the Software under the Agreement.
+
+Contribution: means any or all modifications, corrections, translations,
+adaptations and/or new functions integrated into the Software by any or
+all Contributors, as well as any or all Internal Modules.
+
+Module: means a set of sources files including their documentation that
+enables supplementary functions or services in addition to those offered
+by the Software.
+
+External Module: means any or all Modules, not derived from the
+Software, so that this Module and the Software run in separate address
+spaces, with one calling the other when they are run.
+
+Internal Module: means any or all Module, connected to the Software so
+that they both execute in the same address space.
+
+GNU GPL: means the GNU General Public License version 2 or any
+subsequent version, as published by the Free Software Foundation Inc.
+
+GNU Affero GPL: means the GNU Affero General Public License version 3 or
+any subsequent version, as published by the Free Software Foundation Inc.
+
+EUPL: means the European Union Public License version 1.1 or any
+subsequent version, as published by the European Commission.
+
+Parties: mean both the Licensee and the Licensor.
+
+These expressions may be used both in singular and plural form.
+
+
+    Article 2 - PURPOSE
+
+The purpose of the Agreement is the grant by the Licensor to the
+Licensee of a non-exclusive, transferable and worldwide license for the
+Software as set forth in Article 5 <#scope> hereinafter for the whole
+term of the protection granted by the rights over said Software.
+
+
+    Article 3 - ACCEPTANCE
+
+3.1 The Licensee shall be deemed as having accepted the terms and
+conditions of this Agreement upon the occurrence of the first of the
+following events:
+
+  * (i) loading the Software by any or all means, notably, by
+    downloading from a remote server, or by loading from a physical medium;
+  * (ii) the first time the Licensee exercises any of the rights granted
+    hereunder.
+
+3.2 One copy of the Agreement, containing a notice relating to the
+characteristics of the Software, to the limited warranty, and to the
+fact that its use is restricted to experienced users has been provided
+to the Licensee prior to its acceptance as set forth in Article 3.1
+<#accepting> hereinabove, and the Licensee hereby acknowledges that it
+has read and understood it.
+
+
+    Article 4 - EFFECTIVE DATE AND TERM
+
+
+      4.1 EFFECTIVE DATE
+
+The Agreement shall become effective on the date when it is accepted by
+the Licensee as set forth in Article 3.1 <#accepting>.
+
+
+      4.2 TERM
+
+The Agreement shall remain in force for the entire legal term of
+protection of the economic rights over the Software.
+
+
+    Article 5 - SCOPE OF RIGHTS GRANTED
+
+The Licensor hereby grants to the Licensee, who accepts, the following
+rights over the Software for any or all use, and for the term of the
+Agreement, on the basis of the terms and conditions set forth hereinafter.
+
+Besides, if the Licensor owns or comes to own one or more patents
+protecting all or part of the functions of the Software or of its
+components, the Licensor undertakes not to enforce the rights granted by
+these patents against successive Licensees using, exploiting or
+modifying the Software. If these patents are transferred, the Licensor
+undertakes to have the transferees subscribe to the obligations set
+forth in this paragraph.
+
+
+      5.1 RIGHT OF USE
+
+The Licensee is authorized to use the Software, without any limitation
+as to its fields of application, with it being hereinafter specified
+that this comprises:
+
+ 1. permanent or temporary reproduction of all or part of the Software
+    by any or all means and in any or all form.
+
+ 2. loading, displaying, running, or storing the Software on any or all
+    medium.
+
+ 3. entitlement to observe, study or test its operation so as to
+    determine the ideas and principles behind any or all constituent
+    elements of said Software. This shall apply when the Licensee
+    carries out any or all loading, displaying, running, transmission or
+    storage operation as regards the Software, that it is entitled to
+    carry out hereunder.
+
+
+      5.2 ENTITLEMENT TO MAKE CONTRIBUTIONS
+
+The right to make Contributions includes the right to translate, adapt,
+arrange, or make any or all modifications to the Software, and the right
+to reproduce the resulting software.
+
+The Licensee is authorized to make any or all Contributions to the
+Software provided that it includes an explicit notice that it is the
+author of said Contribution and indicates the date of the creation thereof.
+
+
+      5.3 RIGHT OF DISTRIBUTION
+
+In particular, the right of distribution includes the right to publish,
+transmit and communicate the Software to the general public on any or
+all medium, and by any or all means, and the right to market, either in
+consideration of a fee, or free of charge, one or more copies of the
+Software by any means.
+
+The Licensee is further authorized to distribute copies of the modified
+or unmodified Software to third parties according to the terms and
+conditions set forth hereinafter.
+
+
+        5.3.1 DISTRIBUTION OF SOFTWARE WITHOUT MODIFICATION
+
+The Licensee is authorized to distribute true copies of the Software in
+Source Code or Object Code form, provided that said distribution
+complies with all the provisions of the Agreement and is accompanied by:
+
+ 1. a copy of the Agreement,
+
+ 2. a notice relating to the limitation of both the Licensor's warranty
+    and liability as set forth in Articles 8 and 9,
+
+and that, in the event that only the Object Code of the Software is
+redistributed, the Licensee allows effective access to the full Source
+Code of the Software for a period of at least three years from the
+distribution of the Software, it being understood that the additional
+acquisition cost of the Source Code shall not exceed the cost of the
+data transfer.
+
+
+        5.3.2 DISTRIBUTION OF MODIFIED SOFTWARE
+
+When the Licensee makes a Contribution to the Software, the terms and
+conditions for the distribution of the resulting Modified Software
+become subject to all the provisions of this Agreement.
+
+The Licensee is authorized to distribute the Modified Software, in
+source code or object code form, provided that said distribution
+complies with all the provisions of the Agreement and is accompanied by:
+
+ 1. a copy of the Agreement,
+
+ 2. a notice relating to the limitation of both the Licensor's warranty
+    and liability as set forth in Articles 8 and 9,
+
+and, in the event that only the object code of the Modified Software is
+redistributed,
+
+ 3. a note stating the conditions of effective access to the full source
+    code of the Modified Software for a period of at least three years
+    from the distribution of the Modified Software, it being understood
+    that the additional acquisition cost of the source code shall not
+    exceed the cost of the data transfer.
+
+
+        5.3.3 DISTRIBUTION OF EXTERNAL MODULES
+
+When the Licensee has developed an External Module, the terms and
+conditions of this Agreement do not apply to said External Module, that
+may be distributed under a separate license agreement.
+
+
+        5.3.4 COMPATIBILITY WITH OTHER LICENSES
+
+The Licensee can include a code that is subject to the provisions of one
+of the versions of the GNU GPL, GNU Affero GPL and/or EUPL in the
+Modified or unmodified Software, and distribute that entire code under
+the terms of the same version of the GNU GPL, GNU Affero GPL and/or EUPL.
+
+The Licensee can include the Modified or unmodified Software in a code
+that is subject to the provisions of one of the versions of the GNU GPL,
+GNU Affero GPL and/or EUPL and distribute that entire code under the
+terms of the same version of the GNU GPL, GNU Affero GPL and/or EUPL.
+
+
+    Article 6 - INTELLECTUAL PROPERTY
+
+
+      6.1 OVER THE INITIAL SOFTWARE
+
+The Holder owns the economic rights over the Initial Software. Any or
+all use of the Initial Software is subject to compliance with the terms
+and conditions under which the Holder has elected to distribute its work
+and no one shall be entitled to modify the terms and conditions for the
+distribution of said Initial Software.
+
+The Holder undertakes that the Initial Software will remain ruled at
+least by this Agreement, for the duration set forth in Article 4.2 <#term>.
+
+
+      6.2 OVER THE CONTRIBUTIONS
+
+The Licensee who develops a Contribution is the owner of the
+intellectual property rights over this Contribution as defined by
+applicable law.
+
+
+      6.3 OVER THE EXTERNAL MODULES
+
+The Licensee who develops an External Module is the owner of the
+intellectual property rights over this External Module as defined by
+applicable law and is free to choose the type of agreement that shall
+govern its distribution.
+
+
+      6.4 JOINT PROVISIONS
+
+The Licensee expressly undertakes:
+
+ 1. not to remove, or modify, in any manner, the intellectual property
+    notices attached to the Software;
+
+ 2. to reproduce said notices, in an identical manner, in the copies of
+    the Software modified or not.
+
+The Licensee undertakes not to directly or indirectly infringe the
+intellectual property rights on the Software of the Holder and/or
+Contributors, and to take, where applicable, vis-à-vis its staff, any
+and all measures required to ensure respect of said intellectual
+property rights of the Holder and/or Contributors.
+
+
+    Article 7 - RELATED SERVICES
+
+7.1 Under no circumstances shall the Agreement oblige the Licensor to
+provide technical assistance or maintenance services for the Software.
+
+However, the Licensor is entitled to offer this type of services. The
+terms and conditions of such technical assistance, and/or such
+maintenance, shall be set forth in a separate instrument. Only the
+Licensor offering said maintenance and/or technical assistance services
+shall incur liability therefor.
+
+7.2 Similarly, any Licensor is entitled to offer to its licensees, under
+its sole responsibility, a warranty, that shall only be binding upon
+itself, for the redistribution of the Software and/or the Modified
+Software, under terms and conditions that it is free to decide. Said
+warranty, and the financial terms and conditions of its application,
+shall be subject of a separate instrument executed between the Licensor
+and the Licensee.
+
+
+    Article 8 - LIABILITY
+
+8.1 Subject to the provisions of Article 8.2, the Licensee shall be
+entitled to claim compensation for any direct loss it may have suffered
+from the Software as a result of a fault on the part of the relevant
+Licensor, subject to providing evidence thereof.
+
+8.2 The Licensor's liability is limited to the commitments made under
+this Agreement and shall not be incurred as a result of in particular:
+(i) loss due the Licensee's total or partial failure to fulfill its
+obligations, (ii) direct or consequential loss that is suffered by the
+Licensee due to the use or performance of the Software, and (iii) more
+generally, any consequential loss. In particular the Parties expressly
+agree that any or all pecuniary or business loss (i.e. loss of data,
+loss of profits, operating loss, loss of customers or orders,
+opportunity cost, any disturbance to business activities) or any or all
+legal proceedings instituted against the Licensee by a third party,
+shall constitute consequential loss and shall not provide entitlement to
+any or all compensation from the Licensor.
+
+
+    Article 9 - WARRANTY
+
+9.1 The Licensee acknowledges that the scientific and technical
+state-of-the-art when the Software was distributed did not enable all
+possible uses to be tested and verified, nor for the presence of
+possible defects to be detected. In this respect, the Licensee's
+attention has been drawn to the risks associated with loading, using,
+modifying and/or developing and reproducing the Software which are
+reserved for experienced users.
+
+The Licensee shall be responsible for verifying, by any or all means,
+the suitability of the product for its requirements, its good working
+order, and for ensuring that it shall not cause damage to either persons
+or properties.
+
+9.2 The Licensor hereby represents, in good faith, that it is entitled
+to grant all the rights over the Software (including in particular the
+rights set forth in Article 5 <#scope>).
+
+9.3 The Licensee acknowledges that the Software is supplied "as is" by
+the Licensor without any other express or tacit warranty, other than
+that provided for in Article 9.2 <#good-faith> and, in particular,
+without any warranty as to its commercial value, its secured, safe,
+innovative or relevant nature.
+
+Specifically, the Licensor does not warrant that the Software is free
+from any error, that it will operate without interruption, that it will
+be compatible with the Licensee's own equipment and software
+configuration, nor that it will meet the Licensee's requirements.
+
+9.4 The Licensor does not either expressly or tacitly warrant that the
+Software does not infringe any third party intellectual property right
+relating to a patent, software or any other property right. Therefore,
+the Licensor disclaims any and all liability towards the Licensee
+arising out of any or all proceedings for infringement that may be
+instituted in respect of the use, modification and redistribution of the
+Software. Nevertheless, should such proceedings be instituted against
+the Licensee, the Licensor shall provide it with technical and legal
+expertise for its defense. Such technical and legal expertise shall be
+decided on a case-by-case basis between the relevant Licensor and the
+Licensee pursuant to a memorandum of understanding. The Licensor
+disclaims any and all liability as regards the Licensee's use of the
+name of the Software. No warranty is given as regards the existence of
+prior rights over the name of the Software or as regards the existence
+of a trademark.
+
+
+    Article 10 - TERMINATION
+
+10.1 In the event of a breach by the Licensee of its obligations
+hereunder, the Licensor may automatically terminate this Agreement
+thirty (30) days after notice has been sent to the Licensee and has
+remained ineffective.
+
+10.2 A Licensee whose Agreement is terminated shall no longer be
+authorized to use, modify or distribute the Software. However, any
+licenses that it may have granted prior to termination of the Agreement
+shall remain valid subject to their having been granted in compliance
+with the terms and conditions hereof.
+
+
+    Article 11 - MISCELLANEOUS
+
+
+      11.1 EXCUSABLE EVENTS
+
+Neither Party shall be liable for any or all delay, or failure to
+perform the Agreement, that may be attributable to an event of force
+majeure, an act of God or an outside cause, such as defective
+functioning or interruptions of the electricity or telecommunications
+networks, network paralysis following a virus attack, intervention by
+government authorities, natural disasters, water damage, earthquakes,
+fire, explosions, strikes and labor unrest, war, etc.
+
+11.2 Any failure by either Party, on one or more occasions, to invoke
+one or more of the provisions hereof, shall under no circumstances be
+interpreted as being a waiver by the interested Party of its right to
+invoke said provision(s) subsequently.
+
+11.3 The Agreement cancels and replaces any or all previous agreements,
+whether written or oral, between the Parties and having the same
+purpose, and constitutes the entirety of the agreement between said
+Parties concerning said purpose. No supplement or modification to the
+terms and conditions hereof shall be effective as between the Parties
+unless it is made in writing and signed by their duly authorized
+representatives.
+
+11.4 In the event that one or more of the provisions hereof were to
+conflict with a current or future applicable act or legislative text,
+said act or legislative text shall prevail, and the Parties shall make
+the necessary amendments so as to comply with said act or legislative
+text. All other provisions shall remain effective. Similarly, invalidity
+of a provision of the Agreement, for any reason whatsoever, shall not
+cause the Agreement as a whole to be invalid.
+
+
+      11.5 LANGUAGE
+
+The Agreement is drafted in both French and English and both versions
+are deemed authentic.
+
+
+    Article 12 - NEW VERSIONS OF THE AGREEMENT
+
+12.1 Any person is authorized to duplicate and distribute copies of this
+Agreement.
+
+12.2 So as to ensure coherence, the wording of this Agreement is
+protected and may only be modified by the authors of the License, who
+reserve the right to periodically publish updates or new versions of the
+Agreement, each with a separate number. These subsequent versions may
+address new issues encountered by Free Software.
+
+12.3 Any Software distributed under a given version of the Agreement may
+only be subsequently distributed under the same version of the Agreement
+or a subsequent version, subject to the provisions of Article 5.3.4
+<#compatibility>.
+
+
+    Article 13 - GOVERNING LAW AND JURISDICTION
+
+13.1 The Agreement is governed by French law. The Parties agree to
+endeavor to seek an amicable solution to any disagreements or disputes
+that may arise during the performance of the Agreement.
+
+13.2 Failing an amicable solution within two (2) months as from their
+occurrence, and unless emergency proceedings are necessary, the
+disagreements or disputes shall be referred to the Paris Courts having
+jurisdiction, by the more diligent Party.

Licence_CeCILL_V2-en.txt

@@ -0,0 +1,506 @@
+
+CeCILL FREE SOFTWARE LICENSE AGREEMENT
+
+
+    Notice
+
+This Agreement is a Free Software license agreement that is the result
+of discussions between its authors in order to ensure compliance with
+the two main principles guiding its drafting:
+
+    * firstly, compliance with the principles governing the distribution
+      of Free Software: access to source code, broad rights granted to
+      users,
+    * secondly, the election of a governing law, French law, with which
+      it is conformant, both as regards the law of torts and
+      intellectual property law, and the protection that it offers to
+      both authors and holders of the economic rights over software.
+
+The authors of the CeCILL (for Ce[a] C[nrs] I[nria] L[ogiciel] L[ibre])
+license are:
+
+Commissariat <20> l'Energie Atomique - CEA, a public scientific, technical
+and industrial research establishment, having its principal place of
+business at 25 rue Leblanc, immeuble Le Ponant D, 75015 Paris, France.
+
+Centre National de la Recherche Scientifique - CNRS, a public scientific
+and technological establishment, having its principal place of business
+at 3 rue Michel-Ange, 75794 Paris cedex 16, France.
+
+Institut National de Recherche en Informatique et en Automatique -
+INRIA, a public scientific and technological establishment, having its
+principal place of business at Domaine de Voluceau, Rocquencourt, BP
+105, 78153 Le Chesnay cedex, France.
+
+
+    Preamble
+
+The purpose of this Free Software license agreement is to grant users
+the right to modify and redistribute the software governed by this
+license within the framework of an open source distribution model.
+
+The exercising of these rights is conditional upon certain obligations
+for users so as to preserve this status for all subsequent redistributions.
+
+In consideration of access to the source code and the rights to copy,
+modify and redistribute granted by the license, users are provided only
+with a limited warranty and the software's author, the holder of the
+economic rights, and the successive licensors only have limited liability.
+
+In this respect, the risks associated with loading, using, modifying
+and/or developing or reproducing the software by the user are brought to
+the user's attention, given its Free Software status, which may make it
+complicated to use, with the result that its use is reserved for
+developers and experienced professionals having in-depth computer
+knowledge. Users are therefore encouraged to load and test the
+suitability of the software as regards their requirements in conditions
+enabling the security of their systems and/or data to be ensured and,
+more generally, to use and operate it in the same conditions of
+security. This Agreement may be freely reproduced and published,
+provided it is not altered, and that no provisions are either added or
+removed herefrom.
+
+This Agreement may apply to any or all software for which the holder of
+the economic rights decides to submit the use thereof to its provisions.
+
+
+    Article 1 - DEFINITIONS
+
+For the purpose of this Agreement, when the following expressions
+commence with a capital letter, they shall have the following meaning:
+
+Agreement: means this license agreement, and its possible subsequent
+versions and annexes.
+
+Software: means the software in its Object Code and/or Source Code form
+and, where applicable, its documentation, "as is" when the Licensee
+accepts the Agreement.
+
+Initial Software: means the Software in its Source Code and possibly its
+Object Code form and, where applicable, its documentation, "as is" when
+it is first distributed under the terms and conditions of the Agreement.
+
+Modified Software: means the Software modified by at least one
+Contribution.
+
+Source Code: means all the Software's instructions and program lines to
+which access is required so as to modify the Software.
+
+Object Code: means the binary files originating from the compilation of
+the Source Code.
+
+Holder: means the holder(s) of the economic rights over the Initial
+Software.
+
+Licensee: means the Software user(s) having accepted the Agreement.
+
+Contributor: means a Licensee having made at least one Contribution.
+
+Licensor: means the Holder, or any other individual or legal entity, who
+distributes the Software under the Agreement.
+
+Contribution: means any or all modifications, corrections, translations,
+adaptations and/or new functions integrated into the Software by any or
+all Contributors, as well as any or all Internal Modules.
+
+Module: means a set of sources files including their documentation that
+enables supplementary functions or services in addition to those offered
+by the Software.
+
+External Module: means any or all Modules, not derived from the
+Software, so that this Module and the Software run in separate address
+spaces, with one calling the other when they are run.
+
+Internal Module: means any or all Module, connected to the Software so
+that they both execute in the same address space.
+
+GNU GPL: means the GNU General Public License version 2 or any
+subsequent version, as published by the Free Software Foundation Inc.
+
+Parties: mean both the Licensee and the Licensor.
+
+These expressions may be used both in singular and plural form.
+
+
+    Article 2 - PURPOSE
+
+The purpose of the Agreement is the grant by the Licensor to the
+Licensee of a non-exclusive, transferable and worldwide license for the
+Software as set forth in Article 5 hereinafter for the whole term of the
+protection granted by the rights over said Software. 
+
+
+    Article 3 - ACCEPTANCE
+
+3.1 The Licensee shall be deemed as having accepted the terms and
+conditions of this Agreement upon the occurrence of the first of the
+following events:
+
+    * (i) loading the Software by any or all means, notably, by
+      downloading from a remote server, or by loading from a physical
+      medium;
+    * (ii) the first time the Licensee exercises any of the rights
+      granted hereunder.
+
+3.2 One copy of the Agreement, containing a notice relating to the
+characteristics of the Software, to the limited warranty, and to the
+fact that its use is restricted to experienced users has been provided
+to the Licensee prior to its acceptance as set forth in Article 3.1
+hereinabove, and the Licensee hereby acknowledges that it has read and
+understood it.
+
+
+    Article 4 - EFFECTIVE DATE AND TERM
+
+
+      4.1 EFFECTIVE DATE
+
+The Agreement shall become effective on the date when it is accepted by
+the Licensee as set forth in Article 3.1.
+
+
+      4.2 TERM
+
+The Agreement shall remain in force for the entire legal term of
+protection of the economic rights over the Software.
+
+
+    Article 5 - SCOPE OF RIGHTS GRANTED
+
+The Licensor hereby grants to the Licensee, who accepts, the following
+rights over the Software for any or all use, and for the term of the
+Agreement, on the basis of the terms and conditions set forth hereinafter.
+
+Besides, if the Licensor owns or comes to own one or more patents
+protecting all or part of the functions of the Software or of its
+components, the Licensor undertakes not to enforce the rights granted by
+these patents against successive Licensees using, exploiting or
+modifying the Software. If these patents are transferred, the Licensor
+undertakes to have the transferees subscribe to the obligations set
+forth in this paragraph.
+
+
+      5.1 RIGHT OF USE
+
+The Licensee is authorized to use the Software, without any limitation
+as to its fields of application, with it being hereinafter specified
+that this comprises:
+
+   1. permanent or temporary reproduction of all or part of the Software
+      by any or all means and in any or all form.
+
+   2. loading, displaying, running, or storing the Software on any or
+      all medium.
+
+   3. entitlement to observe, study or test its operation so as to
+      determine the ideas and principles behind any or all constituent
+      elements of said Software. This shall apply when the Licensee
+      carries out any or all loading, displaying, running, transmission
+      or storage operation as regards the Software, that it is entitled
+      to carry out hereunder.
+
+
+      5.2 ENTITLEMENT TO MAKE CONTRIBUTIONS
+
+The right to make Contributions includes the right to translate, adapt,
+arrange, or make any or all modifications to the Software, and the right
+to reproduce the resulting software.
+
+The Licensee is authorized to make any or all Contributions to the
+Software provided that it includes an explicit notice that it is the
+author of said Contribution and indicates the date of the creation thereof.
+
+
+      5.3 RIGHT OF DISTRIBUTION
+
+In particular, the right of distribution includes the right to publish,
+transmit and communicate the Software to the general public on any or
+all medium, and by any or all means, and the right to market, either in
+consideration of a fee, or free of charge, one or more copies of the
+Software by any means.
+
+The Licensee is further authorized to distribute copies of the modified
+or unmodified Software to third parties according to the terms and
+conditions set forth hereinafter.
+
+
+        5.3.1 DISTRIBUTION OF SOFTWARE WITHOUT MODIFICATION
+
+The Licensee is authorized to distribute true copies of the Software in
+Source Code or Object Code form, provided that said distribution
+complies with all the provisions of the Agreement and is accompanied by:
+
+   1. a copy of the Agreement,
+
+   2. a notice relating to the limitation of both the Licensor's
+      warranty and liability as set forth in Articles 8 and 9,
+
+and that, in the event that only the Object Code of the Software is
+redistributed, the Licensee allows future Licensees unhindered access to
+the full Source Code of the Software by indicating how to access it, it
+being understood that the additional cost of acquiring the Source Code
+shall not exceed the cost of transferring the data.
+
+
+        5.3.2 DISTRIBUTION OF MODIFIED SOFTWARE
+
+When the Licensee makes a Contribution to the Software, the terms and
+conditions for the distribution of the resulting Modified Software
+become subject to all the provisions of this Agreement.
+
+The Licensee is authorized to distribute the Modified Software, in
+source code or object code form, provided that said distribution
+complies with all the provisions of the Agreement and is accompanied by:
+
+   1. a copy of the Agreement,
+
+   2. a notice relating to the limitation of both the Licensor's
+      warranty and liability as set forth in Articles 8 and 9,
+
+and that, in the event that only the object code of the Modified
+Software is redistributed, the Licensee allows future Licensees
+unhindered access to the full source code of the Modified Software by
+indicating how to access it, it being understood that the additional
+cost of acquiring the source code shall not exceed the cost of
+transferring the data.
+
+
+        5.3.3 DISTRIBUTION OF EXTERNAL MODULES
+
+When the Licensee has developed an External Module, the terms and
+conditions of this Agreement do not apply to said External Module, that
+may be distributed under a separate license agreement.
+
+
+        5.3.4 COMPATIBILITY WITH THE GNU GPL
+
+The Licensee can include a code that is subject to the provisions of one
+of the versions of the GNU GPL in the Modified or unmodified Software,
+and distribute that entire code under the terms of the same version of
+the GNU GPL.
+
+The Licensee can include the Modified or unmodified Software in a code
+that is subject to the provisions of one of the versions of the GNU GPL,
+and distribute that entire code under the terms of the same version of
+the GNU GPL.
+
+
+    Article 6 - INTELLECTUAL PROPERTY
+
+
+      6.1 OVER THE INITIAL SOFTWARE
+
+The Holder owns the economic rights over the Initial Software. Any or
+all use of the Initial Software is subject to compliance with the terms
+and conditions under which the Holder has elected to distribute its work
+and no one shall be entitled to modify the terms and conditions for the
+distribution of said Initial Software.
+
+The Holder undertakes that the Initial Software will remain ruled at
+least by this Agreement, for the duration set forth in Article 4.2.
+
+
+      6.2 OVER THE CONTRIBUTIONS
+
+The Licensee who develops a Contribution is the owner of the
+intellectual property rights over this Contribution as defined by
+applicable law.
+
+
+      6.3 OVER THE EXTERNAL MODULES
+
+The Licensee who develops an External Module is the owner of the
+intellectual property rights over this External Module as defined by
+applicable law and is free to choose the type of agreement that shall
+govern its distribution.
+
+
+      6.4 JOINT PROVISIONS
+
+The Licensee expressly undertakes:
+
+   1. not to remove, or modify, in any manner, the intellectual property
+      notices attached to the Software;
+
+   2. to reproduce said notices, in an identical manner, in the copies
+      of the Software modified or not.
+
+The Licensee undertakes not to directly or indirectly infringe the
+intellectual property rights of the Holder and/or Contributors on the
+Software and to take, where applicable, vis-<2D>-vis its staff, any and all
+measures required to ensure respect of said intellectual property rights
+of the Holder and/or Contributors.
+
+
+    Article 7 - RELATED SERVICES
+
+7.1 Under no circumstances shall the Agreement oblige the Licensor to
+provide technical assistance or maintenance services for the Software.
+
+However, the Licensor is entitled to offer this type of services. The
+terms and conditions of such technical assistance, and/or such
+maintenance, shall be set forth in a separate instrument. Only the
+Licensor offering said maintenance and/or technical assistance services
+shall incur liability therefor.
+
+7.2 Similarly, any Licensor is entitled to offer to its licensees, under
+its sole responsibility, a warranty, that shall only be binding upon
+itself, for the redistribution of the Software and/or the Modified
+Software, under terms and conditions that it is free to decide. Said
+warranty, and the financial terms and conditions of its application,
+shall be subject of a separate instrument executed between the Licensor
+and the Licensee.
+
+
+    Article 8 - LIABILITY
+
+8.1 Subject to the provisions of Article 8.2, the Licensee shall be
+entitled to claim compensation for any direct loss it may have suffered
+from the Software as a result of a fault on the part of the relevant
+Licensor, subject to providing evidence thereof.
+
+8.2 The Licensor's liability is limited to the commitments made under
+this Agreement and shall not be incurred as a result of in particular:
+(i) loss due the Licensee's total or partial failure to fulfill its
+obligations, (ii) direct or consequential loss that is suffered by the
+Licensee due to the use or performance of the Software, and (iii) more
+generally, any consequential loss. In particular the Parties expressly
+agree that any or all pecuniary or business loss (i.e. loss of data,
+loss of profits, operating loss, loss of customers or orders,
+opportunity cost, any disturbance to business activities) or any or all
+legal proceedings instituted against the Licensee by a third party,
+shall constitute consequential loss and shall not provide entitlement to
+any or all compensation from the Licensor.
+
+
+    Article 9 - WARRANTY
+
+9.1 The Licensee acknowledges that the scientific and technical
+state-of-the-art when the Software was distributed did not enable all
+possible uses to be tested and verified, nor for the presence of
+possible defects to be detected. In this respect, the Licensee's
+attention has been drawn to the risks associated with loading, using,
+modifying and/or developing and reproducing the Software which are
+reserved for experienced users.
+
+The Licensee shall be responsible for verifying, by any or all means,
+the suitability of the product for its requirements, its good working
+order, and for ensuring that it shall not cause damage to either persons
+or properties.
+
+9.2 The Licensor hereby represents, in good faith, that it is entitled
+to grant all the rights over the Software (including in particular the
+rights set forth in Article 5).
+
+9.3 The Licensee acknowledges that the Software is supplied "as is" by
+the Licensor without any other express or tacit warranty, other than
+that provided for in Article 9.2 and, in particular, without any warranty 
+as to its commercial value, its secured, safe, innovative or relevant
+nature.
+
+Specifically, the Licensor does not warrant that the Software is free
+from any error, that it will operate without interruption, that it will
+be compatible with the Licensee's own equipment and software
+configuration, nor that it will meet the Licensee's requirements.
+
+9.4 The Licensor does not either expressly or tacitly warrant that the
+Software does not infringe any third party intellectual property right
+relating to a patent, software or any other property right. Therefore,
+the Licensor disclaims any and all liability towards the Licensee
+arising out of any or all proceedings for infringement that may be
+instituted in respect of the use, modification and redistribution of the
+Software. Nevertheless, should such proceedings be instituted against
+the Licensee, the Licensor shall provide it with technical and legal
+assistance for its defense. Such technical and legal assistance shall be
+decided on a case-by-case basis between the relevant Licensor and the
+Licensee pursuant to a memorandum of understanding. The Licensor
+disclaims any and all liability as regards the Licensee's use of the
+name of the Software. No warranty is given as regards the existence of
+prior rights over the name of the Software or as regards the existence
+of a trademark.
+
+
+    Article 10 - TERMINATION
+
+10.1 In the event of a breach by the Licensee of its obligations
+hereunder, the Licensor may automatically terminate this Agreement
+thirty (30) days after notice has been sent to the Licensee and has
+remained ineffective.
+
+10.2 A Licensee whose Agreement is terminated shall no longer be
+authorized to use, modify or distribute the Software. However, any
+licenses that it may have granted prior to termination of the Agreement
+shall remain valid subject to their having been granted in compliance
+with the terms and conditions hereof.
+
+
+    Article 11 - MISCELLANEOUS
+
+
+      11.1 EXCUSABLE EVENTS
+
+Neither Party shall be liable for any or all delay, or failure to
+perform the Agreement, that may be attributable to an event of force
+majeure, an act of God or an outside cause, such as defective
+functioning or interruptions of the electricity or telecommunications
+networks, network paralysis following a virus attack, intervention by
+government authorities, natural disasters, water damage, earthquakes,
+fire, explosions, strikes and labor unrest, war, etc.
+
+11.2 Any failure by either Party, on one or more occasions, to invoke
+one or more of the provisions hereof, shall under no circumstances be
+interpreted as being a waiver by the interested Party of its right to
+invoke said provision(s) subsequently.
+
+11.3 The Agreement cancels and replaces any or all previous agreements,
+whether written or oral, between the Parties and having the same
+purpose, and constitutes the entirety of the agreement between said
+Parties concerning said purpose. No supplement or modification to the
+terms and conditions hereof shall be effective as between the Parties
+unless it is made in writing and signed by their duly authorized
+representatives.
+
+11.4 In the event that one or more of the provisions hereof were to
+conflict with a current or future applicable act or legislative text,
+said act or legislative text shall prevail, and the Parties shall make
+the necessary amendments so as to comply with said act or legislative
+text. All other provisions shall remain effective. Similarly, invalidity
+of a provision of the Agreement, for any reason whatsoever, shall not
+cause the Agreement as a whole to be invalid.
+
+
+      11.5 LANGUAGE
+
+The Agreement is drafted in both French and English and both versions
+are deemed authentic.
+
+
+    Article 12 - NEW VERSIONS OF THE AGREEMENT
+
+12.1 Any person is authorized to duplicate and distribute copies of this
+Agreement.
+
+12.2 So as to ensure coherence, the wording of this Agreement is
+protected and may only be modified by the authors of the License, who
+reserve the right to periodically publish updates or new versions of the
+Agreement, each with a separate number. These subsequent versions may
+address new issues encountered by Free Software.
+
+12.3 Any Software distributed under a given version of the Agreement may
+only be subsequently distributed under the same version of the Agreement
+or a subsequent version, subject to the provisions of Article 5.3.4.
+
+
+    Article 13 - GOVERNING LAW AND JURISDICTION
+
+13.1 The Agreement is governed by French law. The Parties agree to
+endeavor to seek an amicable solution to any disagreements or disputes
+that may arise during the performance of the Agreement.
+
+13.2 Failing an amicable solution within two (2) months as from their
+occurrence, and unless emergency proceedings are necessary, the
+disagreements or disputes shall be referred to the Paris Courts having
+jurisdiction, by the more diligent Party.
+
+
+Version 2.0 dated 2006-09-05.

Licence_CeCILL_V2-fr.txt

@@ -0,0 +1,512 @@
+
+CONTRAT DE LICENCE DE LOGICIEL LIBRE CeCILL
+
+
+    Avertissement
+
+Ce contrat est une licence de logiciel libre issue d'une concertation
+entre ses auteurs afin que le respect de deux grands principes pr<70>side <20>
+sa r<>daction:
+
+    * d'une part, le respect des principes de diffusion des logiciels
+      libres: acc<63>s au code source, droits <20>tendus conf<6E>r<EFBFBD>s aux
+      utilisateurs,
+    * d'autre part, la d<>signation d'un droit applicable, le droit
+      fran<61>ais, auquel elle est conforme, tant au regard du droit de la
+      responsabilit<69> civile que du droit de la propri<72>t<EFBFBD> intellectuelle
+      et de la protection qu'il offre aux auteurs et titulaires des
+      droits patrimoniaux sur un logiciel.
+
+Les auteurs de la licence CeCILL (pour Ce[a] C[nrs] I[nria] L[ogiciel]
+L[ibre]) sont:
+
+Commissariat <20> l'Energie Atomique - CEA, <20>tablissement public de
+recherche <20> caract<63>re scientifique, technique et industriel, dont le
+si<EFBFBD>ge est situ<74> 25 rue Leblanc, immeuble Le Ponant D, 75015 Paris.
+
+Centre National de la Recherche Scientifique - CNRS, <20>tablissement
+public <20> caract<63>re scientifique et technologique, dont le si<73>ge est
+situ<EFBFBD> 3 rue Michel-Ange, 75794 Paris cedex 16.
+
+Institut National de Recherche en Informatique et en Automatique -
+INRIA, <20>tablissement public <20> caract<63>re scientifique et technologique,
+dont le si<73>ge est situ<74> Domaine de Voluceau, Rocquencourt, BP 105, 78153
+Le Chesnay cedex.
+
+
+    Pr<50>ambule
+
+Ce contrat est une licence de logiciel libre dont l'objectif est de
+conf<EFBFBD>rer aux utilisateurs la libert<72> de modification et de
+redistribution du logiciel r<>gi par cette licence dans le cadre d'un
+mod<EFBFBD>le de diffusion en logiciel libre.
+
+L'exercice de ces libert<72>s est assorti de certains devoirs <20> la charge
+des utilisateurs afin de pr<70>server ce statut au cours des
+redistributions ult<6C>rieures.
+
+L'accessibilit<69> au code source et les droits de copie, de modification
+et de redistribution qui en d<>coulent ont pour contrepartie de n'offrir
+aux utilisateurs qu'une garantie limit<69>e et de ne faire peser sur
+l'auteur du logiciel, le titulaire des droits patrimoniaux et les
+conc<EFBFBD>dants successifs qu'une responsabilit<69> restreinte.
+
+A cet <20>gard l'attention de l'utilisateur est attir<69>e sur les risques
+associ<EFBFBD>s au chargement, <20> l'utilisation, <20> la modification et/ou au
+d<EFBFBD>veloppement et <20> la reproduction du logiciel par l'utilisateur <20>tant
+donn<EFBFBD> sa sp<73>cificit<69> de logiciel libre, qui peut le rendre complexe <20>
+manipuler et qui le r<>serve donc <20> des d<>veloppeurs ou des
+professionnels avertis poss<73>dant des connaissances informatiques
+approfondies. Les utilisateurs sont donc invit<69>s <20> charger et tester
+l'ad<61>quation du logiciel <20> leurs besoins dans des conditions permettant
+d'assurer la s<>curit<69> de leurs syst<73>mes et/ou de leurs donn<6E>es et, plus
+g<EFBFBD>n<EFBFBD>ralement, <20> l'utiliser et l'exploiter dans les m<>mes conditions de
+s<EFBFBD>curit<EFBFBD>. Ce contrat peut <20>tre reproduit et diffus<75> librement, sous
+r<EFBFBD>serve de le conserver en l'<27>tat, sans ajout ni suppression de clauses.
+
+Ce contrat est susceptible de s'appliquer <20> tout logiciel dont le
+titulaire des droits patrimoniaux d<>cide de soumettre l'exploitation aux
+dispositions qu'il contient.
+
+
+    Article 1 - DEFINITIONS
+
+Dans ce contrat, les termes suivants, lorsqu'ils seront <20>crits avec une
+lettre capitale, auront la signification suivante:
+
+Contrat: d<>signe le pr<70>sent contrat de licence, ses <20>ventuelles versions
+post<EFBFBD>rieures et annexes.
+
+Logiciel: d<>signe le logiciel sous sa forme de Code Objet et/ou de Code
+Source et le cas <20>ch<63>ant sa documentation, dans leur <20>tat au moment de
+l'acceptation du Contrat par le Licenci<63>.
+
+Logiciel Initial: d<>signe le Logiciel sous sa forme de Code Source et
+<EFBFBD>ventuellement de Code Objet et le cas <20>ch<63>ant sa documentation, dans
+leur <20>tat au moment de leur premi<6D>re diffusion sous les termes du Contrat.
+
+Logiciel Modifi<66>: d<>signe le Logiciel modifi<66> par au moins une
+Contribution.
+
+Code Source: d<>signe l'ensemble des instructions et des lignes de
+programme du Logiciel et auquel l'acc<63>s est n<>cessaire en vue de
+modifier le Logiciel.
+
+Code Objet: d<>signe les fichiers binaires issus de la compilation du
+Code Source.
+
+Titulaire: d<>signe le ou les d<>tenteurs des droits patrimoniaux d'auteur
+sur le Logiciel Initial.
+
+Licenci<EFBFBD>: d<>signe le ou les utilisateurs du Logiciel ayant accept<70> le
+Contrat.
+
+Contributeur: d<>signe le Licenci<63> auteur d'au moins une Contribution.
+
+Conc<EFBFBD>dant: d<>signe le Titulaire ou toute personne physique ou morale
+distribuant le Logiciel sous le Contrat.
+
+Contribution: d<>signe l'ensemble des modifications, corrections,
+traductions, adaptations et/ou nouvelles fonctionnalit<69>s int<6E>gr<67>es dans
+le Logiciel par tout Contributeur, ainsi que tout Module Interne.
+
+Module: d<>signe un ensemble de fichiers sources y compris leur
+documentation qui permet de r<>aliser des fonctionnalit<69>s ou services
+suppl<EFBFBD>mentaires <20> ceux fournis par le Logiciel.
+
+Module Externe: d<>signe tout Module, non d<>riv<69> du Logiciel, tel que ce
+Module et le Logiciel s'ex<65>cutent dans des espaces d'adressage
+diff<EFBFBD>rents, l'un appelant l'autre au moment de leur ex<65>cution.
+
+Module Interne: d<>signe tout Module li<6C> au Logiciel de telle sorte
+qu'ils s'ex<65>cutent dans le m<>me espace d'adressage.
+
+GNU GPL: d<>signe la GNU General Public License dans sa version 2 ou
+toute version ult<6C>rieure, telle que publi<6C>e par Free Software Foundation
+Inc.
+
+Parties: d<>signe collectivement le Licenci<63> et le Conc<6E>dant.
+
+Ces termes s'entendent au singulier comme au pluriel.
+
+
+    Article 2 - OBJET
+
+Le Contrat a pour objet la concession par le Conc<6E>dant au Licenci<63> d'une
+licence non exclusive, cessible et mondiale du Logiciel telle que
+d<EFBFBD>finie ci-apr<70>s <20> l'article 5 pour toute la dur<75>e de protection des droits
+portant sur ce Logiciel.
+
+
+    Article 3 - ACCEPTATION
+
+3.1 L'acceptation par le Licenci<63> des termes du Contrat est r<>put<75>e
+acquise du fait du premier des faits suivants:
+
+    * (i) le chargement du Logiciel par tout moyen notamment par
+      t<>l<EFBFBD>chargement <20> partir d'un serveur distant ou par chargement <20>
+      partir d'un support physique;
+    * (ii) le premier exercice par le Licenci<63> de l'un quelconque des
+      droits conc<6E>d<EFBFBD>s par le Contrat.
+
+3.2 Un exemplaire du Contrat, contenant notamment un avertissement
+relatif aux sp<73>cificit<69>s du Logiciel, <20> la restriction de garantie et <20>
+la limitation <20> un usage par des utilisateurs exp<78>riment<6E>s a <20>t<EFBFBD> mis <20>
+disposition du Licenci<63> pr<70>alablement <20> son acceptation telle que
+d<EFBFBD>finie <20> l'article 3.1 ci dessus et le Licenci<63> reconna<6E>t en avoir pris
+connaissance.
+
+
+    Article 4 - ENTREE EN VIGUEUR ET DUREE
+
+
+      4.1 ENTREE EN VIGUEUR
+
+Le Contrat entre en vigueur <20> la date de son acceptation par le Licenci<63>
+telle que d<>finie en 3.1.
+
+
+      4.2 DUREE
+
+Le Contrat produira ses effets pendant toute la dur<75>e l<>gale de
+protection des droits patrimoniaux portant sur le Logiciel.
+
+
+    Article 5 - ETENDUE DES DROITS CONCEDES
+
+Le Conc<6E>dant conc<6E>de au Licenci<63>, qui accepte, les droits suivants sur
+le Logiciel pour toutes destinations et pour la dur<75>e du Contrat dans
+les conditions ci-apr<70>s d<>taill<6C>es.
+
+Par ailleurs, si le Conc<6E>dant d<>tient ou venait <20> d<>tenir un ou
+plusieurs brevets d'invention prot<6F>geant tout ou partie des
+fonctionnalit<EFBFBD>s du Logiciel ou de ses composants, il s'engage <20> ne pas
+opposer les <20>ventuels droits conf<6E>r<EFBFBD>s par ces brevets aux Licenci<63>s
+successifs qui utiliseraient, exploiteraient ou modifieraient le
+Logiciel. En cas de cession de ces brevets, le Conc<6E>dant s'engage <20>
+faire reprendre les obligations du pr<70>sent alin<69>a aux cessionnaires.
+
+
+      5.1 DROIT D'UTILISATION
+
+Le Licenci<63> est autoris<69> <20> utiliser le Logiciel, sans restriction quant
+aux domaines d'application, <20>tant ci-apr<70>s pr<70>cis<69> que cela comporte:
+
+   1. la reproduction permanente ou provisoire du Logiciel en tout ou
+      partie par tout moyen et sous toute forme.
+
+   2. le chargement, l'affichage, l'ex<65>cution, ou le stockage du
+      Logiciel sur tout support.
+
+   3. la possibilit<69> d'en observer, d'en <20>tudier, ou d'en tester le
+      fonctionnement afin de d<>terminer les id<69>es et principes qui sont
+      <20> la base de n'importe quel <20>l<EFBFBD>ment de ce Logiciel; et ceci,
+      lorsque le Licenci<63> effectue toute op<6F>ration de chargement,
+      d'affichage, d'ex<65>cution, de transmission ou de stockage du
+      Logiciel qu'il est en droit d'effectuer en vertu du Contrat.
+
+
+      5.2 DROIT D'APPORTER DES CONTRIBUTIONS
+
+Le droit d'apporter des Contributions comporte le droit de traduire,
+d'adapter, d'arranger ou d'apporter toute autre modification au Logiciel
+et le droit de reproduire le logiciel en r<>sultant.
+
+Le Licenci<63> est autoris<69> <20> apporter toute Contribution au Logiciel sous
+r<EFBFBD>serve de mentionner, de fa<66>on explicite, son nom en tant qu'auteur de
+cette Contribution et la date de cr<63>ation de celle-ci.
+
+
+      5.3 DROIT DE DISTRIBUTION
+
+Le droit de distribution comporte notamment le droit de diffuser, de
+transmettre et de communiquer le Logiciel au public sur tout support et
+par tout moyen ainsi que le droit de mettre sur le march<63> <20> titre
+on<EFBFBD>reux ou gratuit, un ou des exemplaires du Logiciel par tout proc<6F>d<EFBFBD>.
+
+Le Licenci<63> est autoris<69> <20> distribuer des copies du Logiciel, modifi<66> ou
+non, <20> des tiers dans les conditions ci-apr<70>s d<>taill<6C>es.
+
+
+        5.3.1 DISTRIBUTION DU LOGICIEL SANS MODIFICATION
+
+Le Licenci<63> est autoris<69> <20> distribuer des copies conformes du Logiciel,
+sous forme de Code Source ou de Code Objet, <20> condition que cette
+distribution respecte les dispositions du Contrat dans leur totalit<69> et
+soit accompagn<67>e:
+
+   1. d'un exemplaire du Contrat,
+
+   2. d'un avertissement relatif <20> la restriction de garantie et de
+      responsabilit<69> du Conc<6E>dant telle que pr<70>vue aux articles 8
+      et 9,
+
+et que, dans le cas o<> seul le Code Objet du Logiciel est redistribu<62>,
+le Licenci<63> permette aux futurs Licenci<63>s d'acc<63>der facilement au Code
+Source complet du Logiciel en indiquant les modalit<69>s d'acc<63>s, <20>tant
+entendu que le co<63>t additionnel d'acquisition du Code Source ne devra
+pas exc<78>der le simple co<63>t de transfert des donn<6E>es.
+
+
+        5.3.2 DISTRIBUTION DU LOGICIEL MODIFIE
+
+Lorsque le Licenci<63> apporte une Contribution au Logiciel, les conditions
+de distribution du Logiciel Modifi<66> en r<>sultant sont alors soumises <20>
+l'int<6E>gralit<69> des dispositions du Contrat.
+
+Le Licenci<63> est autoris<69> <20> distribuer le Logiciel Modifi<66>, sous forme de
+code source ou de code objet, <20> condition que cette distribution
+respecte les dispositions du Contrat dans leur totalit<69> et soit
+accompagn<EFBFBD>e:
+
+   1. d'un exemplaire du Contrat,
+
+   2. d'un avertissement relatif <20> la restriction de garantie et de
+      responsabilit<69> du Conc<6E>dant telle que pr<70>vue aux articles 8
+      et 9,
+
+et que, dans le cas o<> seul le code objet du Logiciel Modifi<66> est
+redistribu<EFBFBD>, le Licenci<63> permette aux futurs Licenci<63>s d'acc<63>der
+facilement au code source complet du Logiciel Modifi<66> en indiquant les
+modalit<EFBFBD>s d'acc<63>s, <20>tant entendu que le co<63>t additionnel d'acquisition
+du code source ne devra pas exc<78>der le simple co<63>t de transfert des donn<6E>es.
+
+
+        5.3.3 DISTRIBUTION DES MODULES EXTERNES
+
+Lorsque le Licenci<63> a d<>velopp<70> un Module Externe les conditions du
+Contrat ne s'appliquent pas <20> ce Module Externe, qui peut <20>tre distribu<62>
+sous un contrat de licence diff<66>rent.
+
+
+        5.3.4 COMPATIBILITE AVEC LA LICENCE GNU GPL
+
+Le Licenci<63> peut inclure un code soumis aux dispositions d'une des
+versions de la licence GNU GPL dans le Logiciel modifi<66> ou non et
+distribuer l'ensemble sous les conditions de la m<>me version de la
+licence GNU GPL.
+
+Le Licenci<63> peut inclure le Logiciel modifi<66> ou non dans un code soumis
+aux dispositions d'une des versions de la licence GNU GPL et distribuer
+l'ensemble sous les conditions de la m<>me version de la licence GNU GPL.
+
+
+    Article 6 - PROPRIETE INTELLECTUELLE
+
+
+      6.1 SUR LE LOGICIEL INITIAL
+
+Le Titulaire est d<>tenteur des droits patrimoniaux sur le Logiciel
+Initial. Toute utilisation du Logiciel Initial est soumise au respect
+des conditions dans lesquelles le Titulaire a choisi de diffuser son
+oeuvre et nul autre n'a la facult<6C> de modifier les conditions de
+diffusion de ce Logiciel Initial.
+
+Le Titulaire s'engage <20> ce que le Logiciel Initial reste au moins r<>gi
+par le Contrat et ce, pour la dur<75>e vis<69>e <20> l'article 4.2.
+
+
+      6.2 SUR LES CONTRIBUTIONS
+
+Le Licenci<63> qui a d<>velopp<70> une Contribution est titulaire sur celle-ci
+des droits de propri<72>t<EFBFBD> intellectuelle dans les conditions d<>finies par
+la l<>gislation applicable.
+
+
+      6.3 SUR LES MODULES EXTERNES
+
+Le Licenci<63> qui a d<>velopp<70> un Module Externe est titulaire sur celui-ci
+des droits de propri<72>t<EFBFBD> intellectuelle dans les conditions d<>finies par
+la l<>gislation applicable et reste libre du choix du contrat r<>gissant
+sa diffusion.
+
+
+      6.4 DISPOSITIONS COMMUNES
+
+Le Licenci<63> s'engage express<73>ment:
+
+   1. <20> ne pas supprimer ou modifier de quelque mani<6E>re que ce soit les
+      mentions de propri<72>t<EFBFBD> intellectuelle appos<6F>es sur le Logiciel;
+
+   2. <20> reproduire <20> l'identique lesdites mentions de propri<72>t<EFBFBD>
+      intellectuelle sur les copies du Logiciel modifi<66> ou non.
+
+Le Licenci<63> s'engage <20> ne pas porter atteinte, directement ou
+indirectement, aux droits de propri<72>t<EFBFBD> intellectuelle du Titulaire et/ou
+des Contributeurs sur le Logiciel et <20> prendre, le cas <20>ch<63>ant, <20>
+l'<27>gard de son personnel toutes les mesures n<>cessaires pour assurer le
+respect des dits droits de propri<72>t<EFBFBD> intellectuelle du Titulaire et/ou
+des Contributeurs.
+
+
+    Article 7 - SERVICES ASSOCIES
+
+7.1 Le Contrat n'oblige en aucun cas le Conc<6E>dant <20> la r<>alisation de
+prestations d'assistance technique ou de maintenance du Logiciel.
+
+Cependant le Conc<6E>dant reste libre de proposer ce type de services. Les
+termes et conditions d'une telle assistance technique et/ou d'une telle
+maintenance seront alors d<>termin<69>s dans un acte s<>par<61>. Ces actes de
+maintenance et/ou assistance technique n'engageront que la seule
+responsabilit<EFBFBD> du Conc<6E>dant qui les propose.
+
+7.2 De m<>me, tout Conc<6E>dant est libre de proposer, sous sa seule
+responsabilit<EFBFBD>, <20> ses licenci<63>s une garantie, qui n'engagera que lui,
+lors de la redistribution du Logiciel et/ou du Logiciel Modifi<66> et ce,
+dans les conditions qu'il souhaite. Cette garantie et les modalit<69>s
+financi<EFBFBD>res de son application feront l'objet d'un acte s<>par<61> entre le
+Conc<EFBFBD>dant et le Licenci<63>.
+
+
+    Article 8 - RESPONSABILITE
+
+8.1 Sous r<>serve des dispositions de l'article 8.2, le Licenci<63> a la 
+facult<EFBFBD>, sous r<>serve de prouver la faute du Conc<6E>dant concern<72>, de
+solliciter la r<>paration du pr<70>judice direct qu'il subirait du fait du
+Logiciel et dont il apportera la preuve.
+
+8.2 La responsabilit<69> du Conc<6E>dant est limit<69>e aux engagements pris en
+application du Contrat et ne saurait <20>tre engag<61>e en raison notamment:
+(i) des dommages dus <20> l'inex<65>cution, totale ou partielle, de ses
+obligations par le Licenci<63>, (ii) des dommages directs ou indirects
+d<EFBFBD>coulant de l'utilisation ou des performances du Logiciel subis par le
+Licenci<EFBFBD> et (iii) plus g<>n<EFBFBD>ralement d'un quelconque dommage indirect. En
+particulier, les Parties conviennent express<73>ment que tout pr<70>judice
+financier ou commercial (par exemple perte de donn<6E>es, perte de
+b<EFBFBD>n<EFBFBD>fices, perte d'exploitation, perte de client<6E>le ou de commandes,
+manque <20> gagner, trouble commercial quelconque) ou toute action dirig<69>e
+contre le Licenci<63> par un tiers, constitue un dommage indirect et
+n'ouvre pas droit <20> r<>paration par le Conc<6E>dant.
+
+
+    Article 9 - GARANTIE
+
+9.1 Le Licenci<63> reconna<6E>t que l'<27>tat actuel des connaissances
+scientifiques et techniques au moment de la mise en circulation du
+Logiciel ne permet pas d'en tester et d'en v<>rifier toutes les
+utilisations ni de d<>tecter l'existence d'<27>ventuels d<>fauts. L'attention
+du Licenci<63> a <20>t<EFBFBD> attir<69>e sur ce point sur les risques associ<63>s au
+chargement, <20> l'utilisation, la modification et/ou au d<>veloppement et <20>
+la reproduction du Logiciel qui sont r<>serv<72>s <20> des utilisateurs avertis.
+
+Il rel<65>ve de la responsabilit<69> du Licenci<63> de contr<74>ler, par tous
+moyens, l'ad<61>quation du produit <20> ses besoins, son bon fonctionnement et
+de s'assurer qu'il ne causera pas de dommages aux personnes et aux biens.
+
+9.2 Le Conc<6E>dant d<>clare de bonne foi <20>tre en droit de conc<6E>der
+l'ensemble des droits attach<63>s au Logiciel (comprenant notamment les
+droits vis<69>s <20> l'article 5).
+
+9.3 Le Licenci<63> reconna<6E>t que le Logiciel est fourni "en l'<27>tat" par le
+Conc<EFBFBD>dant sans autre garantie, expresse ou tacite, que celle pr<70>vue <20>
+l'article 9.2 et notamment sans aucune garantie sur sa valeur commerciale, 
+son caract<63>re s<>curis<69>, innovant ou pertinent.
+
+En particulier, le Conc<6E>dant ne garantit pas que le Logiciel est exempt
+d'erreur, qu'il fonctionnera sans interruption, qu'il sera compatible
+avec l'<27>quipement du Licenci<63> et sa configuration logicielle ni qu'il
+remplira les besoins du Licenci<63>.
+
+9.4 Le Conc<6E>dant ne garantit pas, de mani<6E>re expresse ou tacite, que le
+Logiciel ne porte pas atteinte <20> un quelconque droit de propri<72>t<EFBFBD>
+intellectuelle d'un tiers portant sur un brevet, un logiciel ou sur tout
+autre droit de propri<72>t<EFBFBD>. Ainsi, le Conc<6E>dant exclut toute garantie au
+profit du Licenci<63> contre les actions en contrefa<66>on qui pourraient <20>tre
+diligent<EFBFBD>es au titre de l'utilisation, de la modification, et de la
+redistribution du Logiciel. N<>anmoins, si de telles actions sont
+exerc<EFBFBD>es contre le Licenci<63>, le Conc<6E>dant lui apportera son aide
+technique et juridique pour sa d<>fense. Cette aide technique et
+juridique est d<>termin<69>e au cas par cas entre le Conc<6E>dant concern<72> et
+le Licenci<63> dans le cadre d'un protocole d'accord. Le Conc<6E>dant d<>gage
+toute responsabilit<69> quant <20> l'utilisation de la d<>nomination du
+Logiciel par le Licenci<63>. Aucune garantie n'est apport<72>e quant <20>
+l'existence de droits ant<6E>rieurs sur le nom du Logiciel et sur
+l'existence d'une marque.
+
+
+    Article 10 - RESILIATION
+
+10.1 En cas de manquement par le Licenci<63> aux obligations mises <20> sa
+charge par le Contrat, le Conc<6E>dant pourra r<>silier de plein droit le
+Contrat trente (30) jours apr<70>s notification adress<73>e au Licenci<63> et
+rest<EFBFBD>e sans effet.
+
+10.2 Le Licenci<63> dont le Contrat est r<>sili<6C> n'est plus autoris<69> <20>
+utiliser, modifier ou distribuer le Logiciel. Cependant, toutes les
+licences qu'il aura conc<6E>d<EFBFBD>es ant<6E>rieurement <20> la r<>siliation du Contrat
+resteront valides sous r<>serve qu'elles aient <20>t<EFBFBD> effectu<74>es en
+conformit<EFBFBD> avec le Contrat.
+
+
+    Article 11 - DISPOSITIONS DIVERSES
+
+
+      11.1 CAUSE EXTERIEURE
+
+Aucune des Parties ne sera responsable d'un retard ou d'une d<>faillance
+d'ex<65>cution du Contrat qui serait d<> <20> un cas de force majeure, un cas
+fortuit ou une cause ext<78>rieure, telle que, notamment, le mauvais
+fonctionnement ou les interruptions du r<>seau <20>lectrique ou de
+t<EFBFBD>l<EFBFBD>communication, la paralysie du r<>seau li<6C>e <20> une attaque
+informatique, l'intervention des autorit<69>s gouvernementales, les
+catastrophes naturelles, les d<>g<EFBFBD>ts des eaux, les tremblements de terre,
+le feu, les explosions, les gr<67>ves et les conflits sociaux, l'<27>tat de
+guerre...
+
+11.2 Le fait, par l'une ou l'autre des Parties, d'omettre en une ou
+plusieurs occasions de se pr<70>valoir d'une ou plusieurs dispositions du
+Contrat, ne pourra en aucun cas impliquer renonciation par la Partie
+int<EFBFBD>ress<EFBFBD>e <20> s'en pr<70>valoir ult<6C>rieurement.
+
+11.3 Le Contrat annule et remplace toute convention ant<6E>rieure, <20>crite
+ou orale, entre les Parties sur le m<>me objet et constitue l'accord
+entier entre les Parties sur cet objet. Aucune addition ou modification
+aux termes du Contrat n'aura d'effet <20> l'<27>gard des Parties <20> moins
+d'<27>tre faite par <20>crit et sign<67>e par leurs repr<70>sentants d<>ment habilit<69>s.
+
+11.4 Dans l'hypoth<74>se o<> une ou plusieurs des dispositions du Contrat
+s'av<61>rerait contraire <20> une loi ou <20> un texte applicable, existants ou
+futurs, cette loi ou ce texte pr<70>vaudrait, et les Parties feraient les
+amendements n<>cessaires pour se conformer <20> cette loi ou <20> ce texte.
+Toutes les autres dispositions resteront en vigueur. De m<>me, la
+nullit<EFBFBD>, pour quelque raison que ce soit, d'une des dispositions du
+Contrat ne saurait entra<72>ner la nullit<69> de l'ensemble du Contrat.
+
+
+      11.5 LANGUE
+
+Le Contrat est r<>dig<69> en langue fran<61>aise et en langue anglaise, ces
+deux versions faisant <20>galement foi.
+
+
+    Article 12 - NOUVELLES VERSIONS DU CONTRAT
+
+12.1 Toute personne est autoris<69>e <20> copier et distribuer des copies de
+ce Contrat.
+
+12.2 Afin d'en pr<70>server la coh<6F>rence, le texte du Contrat est prot<6F>g<EFBFBD>
+et ne peut <20>tre modifi<66> que par les auteurs de la licence, lesquels se
+r<EFBFBD>servent le droit de publier p<>riodiquement des mises <20> jour ou de
+nouvelles versions du Contrat, qui poss<73>deront chacune un num<75>ro
+distinct. Ces versions ult<6C>rieures seront susceptibles de prendre en
+compte de nouvelles probl<62>matiques rencontr<74>es par les logiciels libres.
+
+12.3 Tout Logiciel diffus<75> sous une version donn<6E>e du Contrat ne pourra
+faire l'objet d'une diffusion ult<6C>rieure que sous la m<>me version du
+Contrat ou une version post<73>rieure, sous r<>serve des dispositions de
+l'article 5.3.4.
+
+
+    Article 13 - LOI APPLICABLE ET COMPETENCE TERRITORIALE
+
+13.1 Le Contrat est r<>gi par la loi fran<61>aise. Les Parties conviennent
+de tenter de r<>gler <20> l'amiable les diff<66>rends ou litiges qui
+viendraient <20> se produire par suite ou <20> l'occasion du Contrat.
+
+13.2 A d<>faut d'accord amiable dans un d<>lai de deux (2) mois <20> compter
+de leur survenance et sauf situation relevant d'une proc<6F>dure d'urgence,
+les diff<66>rends ou litiges seront port<72>s par la Partie la plus diligente
+devant les Tribunaux comp<6D>tents de Paris.
+
+
+Version 2.0 du 2006-09-05.

README.md

@@ -0,0 +1,40 @@
+The `OBITools3`: A package for the management of analyses and data in DNA metabarcoding   
+---------------------------------------------
+
+
+
+DNA metabarcoding offers new perspectives for biodiversity research [1]. This approach of ecosystem studies relies heavily on the use of Next-Generation Sequencing (NGS), and consequently requires the ability to to treat large volumes of data. The `OBITools` package  satisfies this requirement thanks to a set of programs specifically designed for analyzing NGS data in a DNA metabarcoding context [2] - <http://metabarcoding.org/obitools>. Their capacity to filter and edit sequences while taking into account taxonomic annotation helps to setup tailored-made analysis pipelines for a broad range of DNA metabarcoding applications, including biodiversity surveys or diet analyses.   
+
+
+
+**The `OBITools3`.** This new version of the `OBITools` looks to significantly improve the storage efficiency and the data processing speed. To this end, the `OBITools3` rely on an ad hoc database system, inside which all the data that a DNA metabarcoding experiment must consider is stored: the sequences, the metadata (describing for instance the samples), the database containing the reference sequences used for the taxonomic annotation, as well as the taxonomic databases. Besides the gain in efficiency, this new structure allows an easier access to all the data associated with an experiment.   
+
+
+
+**Column-oriented storage.** An analysis pipeline corresponds to a succession of commands, each computing one step of the analysis, and where the result of the command *n* is used by the command *n+1*. DNA metabarcoding data can easily be represented in the form of tables, and each command can be regarded as an operation transforming one or several 'input' tables into one or several 'output' tables, which can be used by the next command. Many of the basic operations in a pipeline copy without modification an important part of the input tables to the result tables, and use for their calculations only a small part of the input data. In the original `OBITools`, those tables are kept in the form of annotated sequence files in the FASTA or FASTQ format. This has two consequences: i) keeping the transitional results of the analysis pipeline means using disk space for an important volume of redundant data, ii) The coding and decoding of informations that are not actually used represent an important part of the treatment process. The new database system used by the `OBITools3` (called DMS for Data Management System) relies on column-oriented storage. The columns are immutable and can be assembled in views representing the data tables. This way, the data not modified by a command in an input table can easily be associated to the result table without duplicating any information ; and the data not used at all by a command can be associated with the result table without being read. This strategy results in a gain in disk space efficiency by limiting data redundancy, as well as a gain in execution time by limiting data reading, writing and conversion operations. Finally, as a mean to optimize data access, each column is stored in a binary file directly mapped in memory for reading and writing operations.   
+
+
+
+**Storage optimization.** DNA metabarcoding data is intrinsically very redundant. For example, the same sequence corresponding to a species will be present several thousand times across all samples. In order to limit the disk space used and make comparison operations more efficient, data in the form of character strings is stored in columns using a complex indexing structure, efficient on millions of values, coupling hash functions, Bloom filters and AVL trees. Finally, DNA sequences are compressed by encoding each nucleotide on two or four bits depending on whether the sequences contain only the four nucleotides (A, C, G, T) or use the IUPAC codes.   
+
+
+
+**Saving the data processing history.** The totality of the informations used by the `OBITools3` is stored in immutable data structures in the DMS. If a command has to modify a column used as input to produce its result, a new version of that column is created, leaving the initial version intact. This storage system enables to keep, at minimal cost, the totality of the transitional results produced by the pipeline. The storage of metadata describing all the operations that have produced a view (a result table) in the DMS makes possible the creation of an oriented hypergraph, where each node corresponds to a view and each arrow to an operation. By retracing the dependency relationships in this hypergraph, it is possible to rebuild *a posteriori* the entirety of the process that has produced a result table.   
+
+
+
+**Tools.** The `OBITools3` offer the same tools as the original `OBITools`. Eventually, new versions of `ecoPrimers` (PCR primer design) [3], `ecoPCR` (*in silico* PCR) [4], as well as `Sumatra` (sequence alignment) and `Sumaclust` (sequence alignment and clustering) [5] will be added, taking advantage of the database structure developed for the `OBITools3`.    
+
+
+
+**Implementation and disponibility.** The lower layers managing the DMS as well as all the compute-intensive functions are coded in `C99` for efficiency reasons. A `Cython` (<http://www.cython.org>) object layer allows for a simple but efficient implementation of the `OBITools3` commands in `Python 3.5`. The `OBITools3` are still in development, and the first functional versions are expected for autumn 2016.    
+
+
+
+**References.**
+
+1. Taberlet P, Coissac E, Hajibabaei M, Rieseberg LH: Environmental DNA. Mol Ecol 2012:1789–1793.
+2. Boyer F, Mercier C, Bonin A, Le Bras Y, Taberlet P, Coissac E: OBITools: a Unix-inspired software package for DNA metabarcoding. Mol Ecol Resour 2015:n/a–n/a.
+3. Riaz T, Shehzad W, Viari A, Pompanon F, Taberlet P, Coissac E: ecoPrimers: inference of new DNA barcode markers from whole genome sequence analysis. Nucleic Acids Res 2011, 39:e145.
+4. Ficetola GF, Coissac E, Zundel S, Riaz T, Shehzad W, Bessière J, Taberlet P, Pompanon F: An in silico approach for the evaluation of DNA barcodes. BMC Genomics 2010, 11:434.
+5. Mercier C, Boyer F, Bonin A, Coissac E (2013) SUMATRA and SUMACLUST: fast and exact comparison and clustering of sequences. Available: <http://metabarcoding.org/sumatra> and <http://metabarcoding.org/sumaclust>

distutils.ext/obidistutils/command/build_ext.py

@@ -7,107 +7,109 @@ Created on 13 fevr. 2014
 from distutils import log
 import os
 
-from Cython.Distutils import build_ext  as ori_build_ext  # @UnresolvedImport
-
-from Cython.Compiler import Options as cython_options  # @UnresolvedImport
 
 from distutils.errors import DistutilsSetupError
 
-class build_ext(ori_build_ext):
-
+try:
+    from Cython.Distutils import build_ext  as ori_build_ext  # @UnresolvedImport
+    from Cython.Compiler import Options as cython_options  # @UnresolvedImport
+    class build_ext(ori_build_ext):
     
-    def modifyDocScripts(self):
-        build_dir_file=open("doc/sphinx/build_dir.txt","w")
-        print(self.build_lib,file=build_dir_file)
-        build_dir_file.close()
         
-    def initialize_options(self):
-        ori_build_ext.initialize_options(self)  # @UndefinedVariable
-        self.littlebigman = None
-        self.built_files = None
-
-    
-    def finalize_options(self):
-        ori_build_ext.finalize_options(self)  # @UndefinedVariable
-
-        self.set_undefined_options('littlebigman',
-                                   ('littlebigman',  'littlebigman'))
-        
-        self.set_undefined_options('build_files',
-                                   ('files',  'built_files'))
-
-        self.cython_c_in_temp = 1
-                   
-        if self.littlebigman =='-DLITTLE_END':
-            if self.define is None:
-                self.define=[('LITTLE_END',None)]
-            else:
-                self.define.append('LITTLE_END',None)
-        
-    def substitute_sources(self,exe_name,sources):
-        """
-        Substitutes source file name starting by an @ by the actual
-        name of the built file (see --> build_files)
-        """
-        sources = list(sources)
-        for i in range(len(sources)):
-            message = "%s :-> %s" % (exe_name,sources[i])
-            if sources[i][0]=='@':
-                try:
-                    filename = self.built_files[sources[i][1:]]
-                except KeyError:
-                    tmpfilename = os.path.join(self.build_temp,sources[i][1:])
-                    if os.path.isfile       (tmpfilename):
-                        filename = tmpfilename
-                    else:
-                        raise DistutilsSetupError(
-                             'The %s filename declared in the source '
-                             'files of the program %s have not been '
-                             'built by the installation process' % (sources[i],
-                                                                     exe_name))
-                sources[i]=filename
-                log.info("%s changed to %s",message,filename)
-            else:
-                log.info("%s ok",message)
-
-        return sources
-
-    def build_extensions(self):
-        # First, sanity-check the 'extensions' list
-        
-        for ext in self.extensions:
-            ext.sources = self.substitute_sources(ext.name,ext.sources)
+        def modifyDocScripts(self):
+            build_dir_file=open("doc/sphinx/build_dir.txt","w")
+            print(self.build_lib,file=build_dir_file)
+            build_dir_file.close()
             
-        self.check_extensions_list(self.extensions)
-
-        for ext in self.extensions:
-            log.info("%s :-> %s",ext.name,ext.sources)
-            ext.sources = self.cython_sources(ext.sources, ext)
-            self.build_extension(ext)
-
-        
-    def run(self):
-        self.modifyDocScripts()
-
-        for cmd_name in self.get_sub_commands():
-            self.run_command(cmd_name)
-
-        cython_options.annotate = True
-        ori_build_ext.run(self)  # @UndefinedVariable
-        
-
-    def has_files(self):
-        return self.distribution.has_files()
-
-    def has_executables(self):
-        return self.distribution.has_executables()
+        def initialize_options(self):
+            ori_build_ext.initialize_options(self)  # @UndefinedVariable
+            self.littlebigman = None
+            self.built_files = None
     
-    sub_commands = [('build_files',has_files),
-                    ('build_cexe', has_executables)
-                    ] + \
-                   ori_build_ext.sub_commands
-
         
+        def finalize_options(self):
+            ori_build_ext.finalize_options(self)  # @UndefinedVariable
+    
+            self.set_undefined_options('littlebigman',
+                                       ('littlebigman',  'littlebigman'))
+            
+            self.set_undefined_options('build_files',
+                                       ('files',  'built_files'))
+    
+            self.cython_c_in_temp = 1
+                       
+            if self.littlebigman =='-DLITTLE_END':
+                if self.define is None:
+                    self.define=[('LITTLE_END',None)]
+                else:
+                    self.define.append('LITTLE_END',None)
+            
+        def substitute_sources(self,exe_name,sources):
+            """
+            Substitutes source file name starting by an @ by the actual
+            name of the built file (see --> build_files)
+            """
+            sources = list(sources)
+            for i in range(len(sources)):
+                message = "%s :-> %s" % (exe_name,sources[i])
+                if sources[i][0]=='@':
+                    try:
+                        filename = self.built_files[sources[i][1:]]
+                    except KeyError:
+                        tmpfilename = os.path.join(self.build_temp,sources[i][1:])
+                        if os.path.isfile       (tmpfilename):
+                            filename = tmpfilename
+                        else:
+                            raise DistutilsSetupError(
+                                 'The %s filename declared in the source '
+                                 'files of the program %s have not been '
+                                 'built by the installation process' % (sources[i],
+                                                                         exe_name))
+                    sources[i]=filename
+                    log.info("%s changed to %s",message,filename)
+                else:
+                    log.info("%s ok",message)
+    
+            return sources
+    
+        def build_extensions(self):
+            # First, sanity-check the 'extensions' list
+            
+            for ext in self.extensions:
+                ext.sources = self.substitute_sources(ext.name,ext.sources)
+                
+            self.check_extensions_list(self.extensions)
+    
+            for ext in self.extensions:
+                log.info("%s :-> %s",ext.name,ext.sources)
+                ext.sources = self.cython_sources(ext.sources, ext)
+                self.build_extension(ext)
+    
+            
+        def run(self):
+            self.modifyDocScripts()
+    
+            for cmd_name in self.get_sub_commands():
+                self.run_command(cmd_name)
+    
+            cython_options.annotate = True
+            ori_build_ext.run(self)  # @UndefinedVariable
+            
+    
+        def has_files(self):
+            return self.distribution.has_files()
+    
+        def has_executables(self):
+            return self.distribution.has_executables()
+        
+        sub_commands = [('build_files',has_files),
+                        ('build_cexe', has_executables)
+                        ] + \
+                       ori_build_ext.sub_commands
+
+except ImportError:
+    from distutils.command import build_ext  # @UnusedImport
+           
 
 
         

distutils.ext/obidistutils/serenity/bootstrappip.py

@@ -0,0 +1,36 @@
+'''
+Created on 22 janv. 2016
+
+@author: coissac
+'''
+
+import sys
+from urllib import request
+import os.path
+
+from obidistutils.serenity.util import get_serenity_dir
+from obidistutils.serenity.rerun import rerun_with_anothe_python
+from obidistutils.serenity.checkpython import is_a_virtualenv_python
+
+getpipurl="https://bootstrap.pypa.io/get-pip.py"
+
+def bootstrap():
+
+    getpipfile=os.path.join(get_serenity_dir(),"get-pip.py")
+    
+    with request.urlopen(getpipurl) as getpip:
+        with open(getpipfile,"wb") as out:
+            for l in getpip:
+                out.write(l)
+                
+    python = sys.executable
+    
+    if is_a_virtualenv_python():
+        command= "%s %s" % (python,getpipfile)        
+    else:
+        command= "%s %s --user" % (python,getpipfile)
+    
+    os.system(command)
+    
+    rerun_with_anothe_python(python)
+    

distutils.ext/obidistutils/serenity/checkpackage.py

@@ -7,8 +7,13 @@ Created on 2 oct. 2014
 import re
 import os
 
-import pip                                              # @UnresolvedImport
-from pip.utils import get_installed_distributions       # @UnresolvedImport
+try:
+    import pip                                              # @UnresolvedImport
+    from pip.utils import get_installed_distributions       # @UnresolvedImport
+except ImportError:
+    from .bootstrappip import bootstrap
+    bootstrap()
+
 from distutils.version import StrictVersion             # @UnusedImport
 from distutils.errors import DistutilsError
 from distutils import log

distutils.ext/obidistutils/serenity/virtual.py

@@ -59,7 +59,7 @@ def serenity_virtualenv(envname,package,version,minversion='3.4',maxversion=None
                     clear=True, 
                     symlinks=False, 
                     with_pip=True)
-        
+                
         # check the newly created virtualenv
         return serenity_virtualenv(envname,package,version)
     

python/obi.py

@@ -0,0 +1,50 @@
+#!/usr/local/bin/python3.4
+'''
+obi -- shortdesc
+
+obi is a description
+
+It defines classes_and_methods
+
+@author:     user_name
+
+@copyright:  2014 organization_name. All rights reserved.
+
+@license:    license
+
+@contact:    user_email
+@deffield    updated: Updated
+'''
+
+
+default_config = { 'software'       : "The OBITools",
+                   'log'            : False,
+                   'loglevel'       : 'INFO',
+                   'progress'       : True,
+                   'defaultdms'     : None
+                  }
+
+root_config_name='obi'
+
+from obitools3.apps.config import getConfiguration     # @UnresolvedImport
+from obitools3.version import version
+
+__all__     = []
+__version__ = version
+__date__    = '2014-09-28'
+__updated__ = '2014-09-28'
+
+DEBUG = 1
+TESTRUN = 0
+PROFILE = 0
+
+
+if __name__ =="__main__":
+    
+    config = getConfiguration(root_config_name,
+                              default_config)    
+                
+    config[root_config_name]['module'].run(config)
+    
+    
+    

python/obitools3/apps/arguments.pxd

@@ -0,0 +1,3 @@
+#cython: language_level=3
+
+cpdef buildArgumentParser(str configname, str softname)

python/obitools3/apps/arguments.pyx

@@ -0,0 +1,61 @@
+#cython: language_level=3
+
+'''
+Created on 27 mars 2016
+
+@author: coissac
+'''
+
+import argparse
+import sys
+
+
+from .command import getCommandsList
+
+class ObiParser(argparse.ArgumentParser): 
+    def error(self, message):
+        sys.stderr.write('error: %s\n' % message)
+        self.print_help()
+        sys.exit(2)
+
+cpdef buildArgumentParser(str configname, 
+                          str softname):
+    parser = ObiParser()
+    
+    parser.add_argument('--version',   dest='%s:version' % configname, 
+                                       action='store_true', 
+                                       default=False, 
+                        help='Print the version of %s' % softname)
+
+    parser.add_argument('--log',       dest='%s:log' % configname, 
+                                       action='store',
+                                       type=str,
+                                       default=None, 
+                        help='Create a logfile')
+
+    parser.add_argument('--no-progress', dest='%s:progress' % configname, 
+                                       action='store_false', 
+                                       default=None, 
+                        help='Do not print the progress bar during analyzes')
+
+    subparsers = parser.add_subparsers(title='subcommands',
+                                       description='valid subcommands',
+                                       help='additional help')
+    
+    commands = getCommandsList()
+    
+    for c in commands:
+        module = commands[c]
+        
+        if hasattr(module, "run"):
+            if hasattr(module, "__title__"):
+                sub = subparsers.add_parser(c,help=module.__title__)
+            else:
+                sub = subparsers.add_parser(c)
+    
+            if hasattr(module, "addOptions"):
+                module.addOptions(sub)
+            
+            sub.set_defaults(**{'%s:module'  % configname : module})
+                        
+    return parser

python/obitools3/apps/command.pxd

@@ -0,0 +1,3 @@
+#cython: language_level=3
+
+cdef object loadCommand(str name,loader)

python/obitools3/apps/command.pyx

@@ -0,0 +1,44 @@
+#cython: language_level=3
+
+'''
+Created on 27 mars 2016
+
+@author: coissac
+'''
+
+import pkgutil
+
+from obitools3 import commands
+
+cdef object loadCommand(str name,loader):
+    '''
+    Load a command module from its name and an ImpLoader
+    
+    This function is for internal use
+    
+    @param name:   name of the module
+    @type name: str 
+    @param loader: the module loader
+    @type loader: ImpLoader
+    
+    @return the loaded module
+    @rtype: module 
+    '''
+    
+    module = loader.find_module(name).load_module(name)
+    return module
+
+def getCommandsList():
+    '''
+    Returns the list of sub-commands available to the main `obi` command
+    
+    @return: a dict instance with key corresponding to each command and
+             value corresponding to the module
+             
+    @rtype: dict
+    '''
+    
+    cdef dict cmds = dict((x[1],loadCommand(x[1],x[0])) 
+                           for x in pkgutil.iter_modules(commands.__path__) 
+                           if not x[2])
+    return cmds

python/obitools3/apps/config.pxd

@@ -0,0 +1,10 @@
+#cython: language_level=3
+
+cpdef str setRootConfigName(str rootname)
+cpdef str getRootConfigName()
+
+cdef dict buildDefaultConfiguration(str root_config_name,
+                                    dict  config)
+
+cpdef dict getConfiguration(str root_config_name=?,
+                                    dict  config=?)

python/obitools3/apps/config.pyx

@@ -0,0 +1,103 @@
+#cython: language_level=3
+
+'''
+Created on 27 mars 2016
+
+@author: coissac
+'''
+
+import sys
+
+from .command   import  getCommandsList
+from .logging   cimport getLogger
+from .arguments cimport buildArgumentParser
+
+from ..version import version
+from _curses import version
+
+cdef dict __default_config__ = {}
+
+
+cpdef str setRootConfigName(str rootname):
+    global __default_config__
+    if '__root_config__' in __default_config__:
+        if __default_config__["__root_config__"] in __default_config__:
+            __default_config__[rootname]=__default_config__[__default_config__["__root_config__"]]
+            del __default_config__[__default_config__["__root_config__"]]
+    __default_config__['__root_config__']=rootname
+    return rootname
+
+cpdef str getRootConfigName():
+    global __default_config__
+    return __default_config__.get('__root_config__',None)
+
+cdef dict buildDefaultConfiguration(str root_config_name,
+                                    dict  config):
+    global __default_config__
+    
+    __default_config__.clear()
+    setRootConfigName(root_config_name)    
+
+    __default_config__[root_config_name]=config
+    
+    config['version']=version
+    
+    commands = getCommandsList()
+    
+    for c in commands:
+        module = commands[c]
+    
+        assert hasattr(module, "run")
+        
+        if hasattr(module, 'default_config'):
+            __default_config__[c]=module.default_config
+        else:
+            __default_config__[c]={}
+                        
+    return __default_config__
+        
+
+cpdef dict getConfiguration(str root_config_name="__default__",
+                            dict  config={}):
+    global __default_config__
+
+    if '__done__' in __default_config__:
+        return __default_config__
+    
+    if root_config_name=="__default__":
+        raise RuntimeError("No root_config_name specified")
+    
+    if not config:
+        raise RuntimeError("Base configuration is empty")
+    
+    
+    
+    config =  buildDefaultConfiguration(root_config_name,
+                                        config)
+    
+    parser = buildArgumentParser(root_config_name,
+                                 config[root_config_name]['software'])
+    
+    options = vars(parser.parse_args())
+
+    if options['%s:version' % root_config_name]:
+        print("%s - Version %s" % (config[root_config_name]['software'],
+                                   config[root_config_name]['version']))
+        sys.exit(0)
+    
+    for k in options:
+        section,key = k.split(':')
+        s = config[section]
+        if options[k] is not None:
+            s[key]=options[k]
+            
+    if not 'module' in config[root_config_name]:
+        print('\nError: No command specified',file=sys.stderr)
+        parser.print_help()
+        sys.exit(2)
+                
+    getLogger(config)
+    
+    config['__done__']=True
+            
+    return config

python/obitools3/apps/logging.pxd

@@ -0,0 +1,3 @@
+#cython: language_level=3
+
+cpdef getLogger(dict config)

python/obitools3/apps/logging.pyx

@@ -0,0 +1,46 @@
+#cython: language_level=3
+
+'''
+Created on 27 mars 2016
+
+@author: coissac
+'''
+
+import logging
+import sys
+
+cpdef getLogger(dict config):
+    '''
+    Returns the logger as defined by the command line option
+    or by the config file
+    :param config:
+    '''
+    
+    root = config["__root_config__"]
+ 
+    level  = config[root]['loglevel'] 
+    logfile= config[root]['log'] 
+    
+    rootlogger   = logging.getLogger()
+    logFormatter = logging.Formatter("%(asctime)s [%(levelname)-5.5s]  %(message)s")
+
+    stderrHandler = logging.StreamHandler(sys.stderr)
+    stderrHandler.setFormatter(logFormatter)
+
+    rootlogger.addHandler(stderrHandler)
+    
+    if logfile:
+        fileHandler = logging.FileHandler(logfile)
+        fileHandler.setFormatter(logFormatter)
+        rootlogger.addHandler(fileHandler)
+    
+    try:
+        loglevel = getattr(logging, level) 
+    except:
+        loglevel = logging.INFO
+        
+    rootlogger.setLevel(loglevel)
+    
+    config[root]['logger']=rootlogger
+    
+    return rootlogger

python/obitools3/apps/progress.pxd

@@ -0,0 +1,65 @@
+#cython: language_level=3
+
+from ..utils cimport str2bytes
+
+cdef extern from "stdio.h":
+    struct FILE
+    int fprintf(FILE *stream, char *format, ...)
+    FILE* stderr
+    ctypedef unsigned int off_t "unsigned long long"
+    
+cdef extern from "unistd.h":
+    int fsync(int fd);
+
+cdef extern from "time.h":
+    struct tm :
+        int tm_yday 
+        int tm_hour
+        int tm_min
+        int tm_sec
+
+    enum: CLOCKS_PER_SEC
+    
+    ctypedef int time_t
+    ctypedef int clock_t
+    ctypedef int suseconds_t
+    
+    struct timeval:
+        time_t      tv_sec     #  seconds */
+        suseconds_t tv_usec    #  microseconds */
+
+
+    struct timezone :
+        int tz_minuteswest;    # minutes west of Greenwich
+        int tz_dsttime;        # type of DST correction 
+
+        
+    int gettimeofday(timeval *tv, timezone *tz)
+
+
+    tm *gmtime_r(time_t *clock, tm *result)
+    time_t time(time_t *tloc)
+    clock_t clock()
+
+cdef class ProgressBar:
+    cdef off_t   maxi
+    cdef clock_t starttime
+    cdef clock_t lasttime
+    cdef clock_t tickcount
+    cdef int freq
+    cdef int cycle
+    cdef int arrow
+    cdef int lastlog
+    cdef bint ontty
+    cdef int fd
+    
+    cdef bytes head
+    cdef char *chead
+    
+    cdef object logger
+
+    cdef char  *wheel
+    cdef char  *spaces
+    cdef char*  diese 
+    
+    cdef clock_t clock(self)

python/obitools3/apps/progress.pyx

@@ -0,0 +1,138 @@
+#cython: language_level=3
+
+'''
+Created on 27 mars 2016
+
+@author: coissac
+'''
+
+import sys
+from ..utils cimport bytes2str
+
+cdef class ProgressBar:
+    cdef clock_t clock(self):
+        cdef clock_t t
+        cdef timeval tp
+        cdef clock_t s
+        
+        <void> gettimeofday(&tp,NULL)
+        s = <clock_t> (<double> tp.tv_usec * 1.e-6 * <double> CLOCKS_PER_SEC)
+        t = tp.tv_sec * CLOCKS_PER_SEC + s 
+        
+        return t
+
+    def __init__(self,
+                 off_t maxi,
+                 dict  config,
+                 str head="",
+                 double seconde=0.1):
+        self.starttime = self.clock()
+        self.lasttime  = self.starttime
+        self.tickcount = <clock_t> (seconde * CLOCKS_PER_SEC)
+        self.freq      = 1
+        self.cycle     = 0
+        self.arrow     = 0
+        self.lastlog   = 0
+        
+        self.ontty = sys.stderr.isatty()
+        
+        if (maxi<=0):
+            maxi=1
+            
+        self.maxi = maxi
+        self.head = str2bytes(head)
+        self.chead= self.head 
+        
+        
+        self.logger=config[config["__root_config__"]]["logger"]
+        self.wheel =  '|/-\\'
+        self.spaces='          ' \
+                    '          ' \
+                    '          ' \
+                    '          ' \
+                    '          '
+        self.diese ='##########' \
+                    '##########' \
+                    '##########' \
+                    '##########' \
+                    '##########'  
+                           
+    def __call__(self,object pos):
+        cdef off_t    ipos
+        cdef clock_t  elapsed
+        cdef clock_t  newtime
+        cdef clock_t  delta
+        cdef clock_t  more 
+        cdef double   percent 
+        cdef tm remain
+        cdef int days,hour,minu,sec
+        cdef off_t fraction
+        cdef int twentyth
+        
+        self.cycle+=1
+    
+        if self.cycle % self.freq == 0:
+            self.cycle=1
+            newtime  = self.clock()
+            delta         = newtime - self.lasttime
+            self.lasttime = newtime
+            elapsed       = newtime - self.starttime
+#            print(" ",delta,elapsed,elapsed/CLOCKS_PER_SEC,self.tickcount)
+            
+            if   delta < self.tickcount / 5 :
+                self.freq*=2
+            elif delta > self.tickcount * 5 and self.freq>1:
+                self.freq/=2
+                
+            
+            if callable(pos):
+                ipos=pos()
+            else:
+                ipos=pos
+                
+            if ipos==0:
+                ipos=1                
+
+            percent = <double>ipos/<double>self.maxi
+            more = <time_t>((<double>elapsed / percent * (1. - percent))/CLOCKS_PER_SEC)
+            <void>gmtime_r(&more, &remain)
+            days  = remain.tm_yday 
+            hour  = remain.tm_hour
+            minu  = remain.tm_min
+            sec   = remain.tm_sec
+                
+            if self.ontty:
+                fraction=<int>(percent * 50.)
+                self.arrow=(self.arrow+1) % 4
+                self.diese[fraction]=0
+                self.spaces[50 - fraction]=0
+            
+                if days:
+                    <void>fprintf(stderr,b'\r%s %5.1f %% |%s%c%s] remain : %d days %02d:%02d:%02d',
+                                    self.chead,
+                                    percent*100,
+                                    self.diese,self.wheel[self.arrow],self.spaces,
+                                    days,hour,minu,sec)
+                else:
+                    <void>fprintf(stderr,b'\r%s %5.1f %% |%s%c%s] remain : %02d:%02d:%02d',
+                                    self.chead,
+                                    percent*100.,
+                                    self.diese,self.wheel[self.arrow],self.spaces,
+                                    hour,minu,sec)
+                self.diese[fraction]=b'#'
+                self.spaces[50 - fraction]=b' '
+
+            twentyth = int(percent * 20)
+            if twentyth != self.lastlog:
+                
+                if self.ontty:
+                    <void>fprintf(stderr,b'\n')
+                    
+                self.logger.info('%s %5.1f %% remain : %02d:%02d:%02d' % (
+                                        bytes2str(self.head),
+                                        percent*100.,
+                                        hour,minu,sec))
+                self.lastlog=twentyth
+        else:
+            self.cycle+=1
+

python/obitools3/commands/count.py

@@ -0,0 +1,44 @@
+'''
+Created on 8 mars 2016
+
+@author: coissac
+'''
+
+from obitools3.apps.progress import ProgressBar  # @UnresolvedImport
+import time
+
+__title__="Counts sequences in a sequence set"
+
+
+default_config = {   'countmode'       : None
+                 }
+
+def addOptions(parser):
+    parser.add_argument(dest='obi:input',     metavar='obi:input', 
+                                              nargs='?', 
+                                              default=None,
+                        help='input data set' )
+
+    group=parser.add_argument_group('Obicount specific options')
+    group.add_argument('-s','--sequence',
+                             action="store_true", dest="count:sequence",
+                             default=False,
+                             help="Prints only the number of sequence records."
+                             )
+ 
+    group.add_argument('-a','--all',
+                             action="store_true", dest="count:all",
+                             default=False,
+                             help="Prints only the total count of sequence records (if a sequence has no `count` attribute, its default count is 1) (default: False)."
+                             )
+
+    
+    
+def run(config):
+    # The code of my command
+    pb = ProgressBar(1000,config,seconde=1)
+    
+    for i in range(1,1001):
+        pb(i)
+        time.sleep(0.01)
+        

python/obitools3/commands/import.pyx

@@ -0,0 +1,136 @@
+from obitools3.apps.progress cimport ProgressBar  # @UnresolvedImport
+from obitools3.files.universalopener cimport uopen
+from obitools3.parsers.fasta import fastaIterator
+from obitools3.parsers.fastq import fastqIterator
+from obitools3.obidms._obidms import OBIDMS
+
+
+import time
+
+__title__="Counts sequences in a sequence set"
+
+
+default_config = {   'destview'     : None,
+                     'skip'         : 0,
+                     'only'         : None,
+                     'skiperror'    : False,
+                     'seqinformat'  : None,
+                     'moltype'      : 'nuc',
+                     'filename'     : None
+                 }
+
+def addOptions(parser):
+    parser.add_argument(dest='import:filename',     
+                        metavar='<FILENAME>', 
+                        nargs='?', 
+                        default=None,
+                        help='sequence file name to be imported' )
+
+    group=parser.add_argument_group('obi import specific options')
+
+    group.add_argument('--default-dms','-d',
+                     action="store", dest="obi:defaultdms",
+                     metavar='<DMS NAME>',
+                     default=None,
+                     type=str,
+                     help="Name of the default DMS for reading and writing data")
+    
+    
+    group.add_argument('--destination-view','-v',
+                     action="store", dest="import:destview",
+                     metavar='<VIEW NAME>',
+                     default=None,
+                     type=str,
+                     required=True,
+                     help="Name of the default DMS for reading and writing data")
+
+    group=parser.add_argument_group('obi import specific options')
+    
+    group.add_argument('--skip',
+                     action="store", dest="import:skip",
+                     metavar='<N>',
+                     default=None,
+                     type=int,
+                     help="skip the N first sequences")
+
+    group.add_argument('--only',
+                     action="store", dest="import:only",
+                     metavar='<N>',
+                     default=None,
+                     type=int,
+                     help="treat only N sequences")
+
+    group.add_argument('--skip-on-error',
+                     action="store_true", dest="import:skiperror",
+                     default=None,
+                     help="Skip sequence entries with parse error")
+    
+    group.add_argument('--fasta',
+                     action="store_const", dest="import:seqinformat",
+                     default=None,
+                     const='fasta',
+                     help="Input file is in fasta nucleic format (including obitools fasta extentions)")
+
+    group.add_argument('--fastq',
+                     action="store_const", dest="import:seqinformat",
+                     default=None,
+                     const='fastq',
+                     help="Input file is in sanger fastq nucleic format (standard fastq)")
+
+    group.add_argument('--nuc',
+                     action="store_const", dest="import:moltype",
+                     default=None,
+                     const='nuc',
+                     help="Input file contains nucleic sequences")
+    
+    group.add_argument('--prot',
+                     action="store_const", dest="import:moltype",
+                     default=None,
+                     const='pep',
+                     help="Input file contains protein sequences")
+
+    
+    
+def run(config):
+    pb = ProgressBar(35000000,config,seconde=5)
+      
+    inputs = uopen(config['import']['filename'])
+    
+    if config['import']['seqinformat']=='fasta':
+        iseq = fastaIterator(inputs)
+        view_type="NUC_SEQS_VIEW"
+    elif config['import']['seqinformat']=='fastq':
+        iseq = fastqIterator(inputs)
+        view_type="NUC_SEQS_VIEW"
+    else:
+        raise RuntimeError('No file format specified')
+    
+    # Temporary way to handle NA values
+    #NA_list = ["nan"]
+    
+    # Create DMS
+    d = OBIDMS(config['obi']['defaultdms'])
+    
+    # Create view
+    view = d.new_view(config['import']['destview'], view_type=view_type)
+    
+    i = 0
+    for seq in iseq:
+        pb(i)
+        view[i].set_id(seq['id'])
+        view[i].set_definition(seq['definition'])
+        view[i].set_sequence(seq['sequence'])
+        for tag in seq['tags'] :
+            #print(tag, seq['tags'][tag])
+            #if seq['tags'][tag] not in NA_list :
+            view[i][tag] = seq['tags'][tag]
+        i+=1
+
+   #print(view)
+    print(view.__repr__())
+
+    view.save_and_close()
+    d.close()
+    
+    print("Done.")
+        

python/obitools3/files/linebuffer.pxd

@@ -0,0 +1,6 @@
+#cython: language_level=3
+
+
+cdef class LineBuffer:
+    cdef object fileobj
+    cdef int    size

python/obitools3/files/linebuffer.pyx

@@ -0,0 +1,23 @@
+#cython: language_level=3
+
+'''
+Created on 30 mars 2016
+
+@author: coissac
+'''
+
+cdef class LineBuffer:
+
+    def __init__(self,object fileobj,int size=100000000):
+        self.fileobj=fileobj 
+        self.size=size
+    
+    def __iter__(self):
+        cdef list buff = self.fileobj.readlines(self.size)
+        cdef str  l
+        
+        while buff:
+            for l in buff:
+                yield l
+            buff = self.fileobj.readlines(self.size)
+    

python/obitools3/files/uncompress.pxd

@@ -0,0 +1,17 @@
+#cython: language_level=3
+
+cdef class MagicKeyFile:
+    cdef object stream
+    cdef str    stream_mode
+    cdef object binary
+    cdef bytes  key 
+    cdef int    keylength 
+    cdef int    pos
+    
+    cpdef bytes read(self,int size=?)
+    cpdef int tell(self)
+
+    
+cdef class CompressedFile:
+    cdef object accessor
+    

python/obitools3/files/uncompress.pyx

@@ -0,0 +1,114 @@
+#cython: language_level=3
+
+'''
+Created on 28 mars 2016
+
+@author: coissac
+'''
+
+import zipfile
+import bz2
+import gzip
+
+import io
+
+cdef class MagicKeyFile:
+    def __init__(self,stream,length=2):
+        
+        binary=stream
+        self.stream = stream
+        self.stream_mode = None 
+        if hasattr(stream, "mode"):
+            self.stream_mode = stream.mode
+            if (not 'b' in stream.mode and 
+                hasattr(stream, "buffer") and 
+                'b' in stream.buffer.mode):
+                binary=stream.buffer
+
+        if (self.stream_mode is None and 
+            not (hasattr(stream, 'headers') and 
+                 hasattr(stream.headers, "keys") and
+                 'Content-type' in stream.headers)):
+            raise TypeError("stream does not present the good interface")
+        
+        self.binary=binary
+        self.key=binary.read(length)
+        self.keylength=length
+        self.pos=0
+        
+    cpdef bytes read(self,int size=-1):
+        cdef bytes r
+        
+        if self.pos < self.keylength:
+            if size > (self.keylength - self.pos):
+                size = size - self.keylength + self.pos
+                r = self.key[self.pos:] + self.binary.read(size)
+                self.pos=self.keylength + 1
+            elif size >=0 :
+                r = self.key[self.pos:(self.pos+size)]
+                self.pos+=size
+            else:
+                r = self.key[self.pos:] + self.binary.read(size)
+                self.pos=self.keylength + 1
+        else:
+            r = self.binary.read(size)
+            
+        return r
+    
+    cpdef int tell(self):
+        cdef int p
+        
+        if self.pos < self.keylength:
+            p = self.pos 
+        else:
+            p = self.tell()
+        
+        return p
+    
+    def __getattr__(self,name):
+        return getattr(self.binary, name)
+        
+
+    
+cdef class CompressedFile:
+    
+    
+    def __init__(self,stream):
+        cdef int            keylength
+        cdef MagicKeyFile   magic
+        cdef str            compressor
+        cdef bytes          k 
+        cdef object         c
+        
+        cdef dict compress = { 'zip' : (b'\x50\x4b\x03\x04',zipfile.ZipFile),
+                               'bz2' : (b'\x42\x5a\x68',bz2.BZ2File),
+                               'gz'  : (b'\x1f\x8b\x08',gzip.open)
+                   }
+        
+        keylength = max([len(x[0]) for x in compress.values()])
+        magic=MagicKeyFile(stream,keylength)
+        
+        self.accessor = None
+        
+        for compressor in compress:
+            k,c = compress[compressor]
+            if magic.key.startswith(k):
+                self.accessor = c(magic)
+        
+        if self.accessor is None:
+            self.accessor = magic
+        
+        if ((hasattr(stream, 'headers') and 
+             hasattr(stream.headers, "keys") and
+             'Content-type' in stream.headers and
+             stream.headers['Content-type'].startswith('text/')) or
+            'b' not in magic.stream_mode):
+            self.accessor = io.TextIOWrapper(self.accessor)
+        
+    
+    def __getattr__(self,name):
+        return getattr(self.accessor, name)
+    
+    def __iter__(self):
+        for x in self.accessor:
+            yield x 

python/obitools3/files/universalopener.pxd

@@ -0,0 +1,5 @@
+#cython: language_level=3
+
+from .uncompress cimport CompressedFile
+
+cpdef CompressedFile uopen(str name, mode=?)

python/obitools3/files/universalopener.pyx

@@ -0,0 +1,23 @@
+#cython: language_level=3
+
+'''
+Created on 25 mars 2016
+
+@author: coissac
+'''
+
+from urllib.request import urlopen
+
+
+cpdef CompressedFile uopen(str name, mode='r'):
+    cdef CompressedFile c
+    
+    try:
+        f = urlopen(name)
+    except:
+        f = open(name,mode) 
+        
+    c = CompressedFile(f)
+    
+    return c
+    

python/obitools3/obidms/_obidms.cfiles

@@ -1,7 +1,28 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
 ../../../src/obidms.h
 ../../../src/obidms.c
-../../../src/obiview.h
-../../../src/obiview.c
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
@@ -12,13 +33,19 @@
 ../../../src/obilittlebigman.c
 ../../../src/obitypes.h
 ../../../src/obitypes.c
-../../../src/private_at_functions.h
-../../../src/private_at_functions.c
-../../../src/obiavl.h
-../../../src/obiavl.c
-../../../src/encode.h
-../../../src/encode.c
-../../../src/obidmscolumn_idx.h
-../../../src/obidmscolumn_idx.c
-../../../src/obidms_taxonomy.c
-../../../src/obidms_taxonomy.h
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/utils.h
+../../../src/utils.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h

python/obitools3/obidms/_obidms.pxd

@@ -45,7 +45,6 @@ cdef class OBIView:
     cdef str       name
     cdef str       comments
     cdef dict      columns
-    cdef dict      columns_pp    # TODO this dict might be unnecessary
     cdef OBIDMS    dms
     
     cpdef delete_column(self, str column_name)
@@ -56,7 +55,7 @@ cdef class OBIView:
                      index_t nb_lines=*, 
                      index_t nb_elements_per_line=*,
                      list elements_names=*,
-                     str avl_name=*,
+                     str indexer_name=*,
                      str comments=*,
                      bint create=*
                     )
@@ -69,7 +68,7 @@ cdef class OBIView_NUC_SEQS(OBIView):
 
     cdef OBIDMS_column ids
     cdef OBIDMS_column sequences
-    cdef OBIDMS_column descriptions
+    cdef OBIDMS_column definitions
     
     cpdef delete_column(self, str column_name)
     

python/obitools3/obidms/_obidms.pyx

@@ -5,10 +5,11 @@ from obitools3.utils cimport bytes2str, str2bytes
 from .capi.obidms cimport obi_dms, \
                           obi_close_dms
                           
-from .capi.obidmscolumn cimport obi_truncate_and_close_column, \
-                                obi_column_format_date, \
+from .capi.obidmscolumn cimport obi_close_column, \
                                 OBIDMS_column_p, \
                                 OBIDMS_column_header_p
+
+from .capi.obiutils cimport obi_format_date
                                 
 from .capi.obitypes cimport const_char_p, \
                             OBIType_t, \
@@ -57,8 +58,8 @@ from .capi.obiview cimport Obiview_p, \
                            obi_new_view_cloned_from_name, \
                            obi_new_view_nuc_seqs_cloned_from_name, \
                            obi_open_view,  \
-                           obi_read_views, \
-                           obi_unmap_read_views, \
+                           obi_read_view_infos, \
+                           obi_close_view_infos, \
                            obi_view_delete_column, \
                            obi_view_add_column, \
                            obi_view_get_column, \
@@ -69,7 +70,7 @@ from .capi.obiview cimport Obiview_p, \
                            VIEW_TYPE_NUC_SEQS, \
                            NUC_SEQUENCE_COLUMN, \
                            ID_COLUMN, \
-                           DESCRIPTION_COLUMN
+                           DEFINITION_COLUMN
                            
 from libc.stdlib cimport malloc
 from cpython.pycapsule cimport PyCapsule_New, PyCapsule_GetPointer
@@ -83,13 +84,13 @@ cdef class OBIDMS_column :
         cdef OBIDMS_column_p  column_p
         cdef OBIDMS_column_p* column_pp
         
-        column_pp = <OBIDMS_column_p*> PyCapsule_GetPointer(((view.columns_pp)[column_name]), NULL) # or use C function
+        column_pp = obi_view_get_pointer_on_column_in_view(view.pointer, str2bytes(column_name))
         column_p = column_pp[0]     # TODO ugly cython dereferencing but can't find better
         
         # Fill structure
         self.pointer = column_pp
         self.dms = view.dms
-        self.view = view.pointer    # TODO pointer or instance?
+        self.view = view.pointer
         self.data_type = bytes2str(name_data_type((column_p.header).returned_data_type))
         self.column_name = bytes2str((column_p.header).name)
         self.nb_elements_per_line = (column_p.header).nb_elements_per_line
@@ -117,6 +118,9 @@ cdef class OBIDMS_column :
         lines_used = (self.pointer)[0].header.lines_used
         for line_nb in range(lines_used):
             yield self.get_line(line_nb)
+
+    def __richcmp__(self, object value, int comp):
+        return self.compare(value, comp)
     
     cpdef update_pointer(self):
         self.pointer = <OBIDMS_column_p*> obi_view_get_pointer_on_column_in_view(self.view, str2bytes(self.column_name))
@@ -131,20 +135,23 @@ cdef class OBIDMS_column :
         return (self.pointer)[0].header.lines_used
         
     cpdef str get_creation_date(self):
-        return bytes2str(obi_column_format_date((self.pointer)[0].header.creation_date))
+        return bytes2str(obi_format_date((self.pointer)[0].header.creation_date))
     
     cpdef str get_comments(self):
         return bytes2str((self.pointer)[0].header.comments)
     
-    def __repr__(self) :
+    def __str__(self) :
         cdef str to_print
         to_print = ''
         for line in self :
             to_print = to_print + str(line) + "\n"
         return to_print
+
+    def __repr__(self) :
+        return (self.column_name + ", version " + str((self.pointer)[0].header.version) + ", data type: " + self.data_type)
         
     cpdef close(self):
-        if obi_truncate_and_close_column((self.pointer)[0]) < 0 :
+        if obi_close_column((self.pointer)[0]) < 0 :
             raise Exception("Problem closing a column")
 
 
@@ -236,27 +243,27 @@ cdef class OBIView :
     
     def __init__(self, OBIDMS dms, str view_name, bint new=False, object view_to_clone=None, list line_selection=None, str comments=""): 
         
-        cdef Obiview_p view = NULL
-        cdef int i
-        cdef list col_list
-        cdef str col_name
-        cdef OBIDMS_column column
-        cdef OBIDMS_column_p column_p
-        cdef OBIDMS_column_p* column_pp
+        cdef Obiview_p              view = NULL
+        cdef int                    i
+        cdef list                   col_list
+        cdef str                    col_name
+        cdef OBIDMS_column          column
+        cdef OBIDMS_column_p        column_p
         cdef OBIDMS_column_header_p header
-        cdef index_t* line_selection_p
-        cdef object col_capsule
+        cdef index_t*               line_selection_p
         
         self.dms = dms
         
+        # Create the C array for the line selection if needed
         if line_selection is not None :
             line_selection_p = <index_t*> malloc((len(line_selection) + 1) * sizeof(index_t))
             for i in range(len(line_selection)) :
-                line_selection_p[i] = line_selection[i]     # TODO type problem?
+                line_selection_p[i] = line_selection[i]
             line_selection_p[len(line_selection)] = -1
         else :
             line_selection_p = NULL
        
+        # Create the view if needed
         if new :
             if view_to_clone is not None :
                 if type(view_to_clone) == str :
@@ -265,50 +272,36 @@ cdef class OBIView :
                     view = obi_new_view(dms.pointer, str2bytes(view_name), (<OBIView> view_to_clone).pointer, line_selection_p, str2bytes(comments))
             elif view_to_clone is None :
                 view = obi_new_view(dms.pointer, str2bytes(view_name), NULL, line_selection_p, str2bytes(comments))
+        # Else, open the existing view
         elif not new :
             if view_name is not None :
                 view = obi_open_view(dms.pointer, str2bytes(view_name))
             elif view_name is None :
-                view = obi_open_view(dms.pointer, NULL)
+                view = obi_open_view(dms.pointer, NULL) # TODO discuss
         
         if view == NULL :
-            raise Exception("Error creating/opening view")
+            raise Exception("Error creating/opening a view")
 
         self.pointer = view
         self.name = bytes2str(view.name)
         
-        # go through columns to build list and open python object (TODO make separate function?)
+        # Go through columns to build list of corresponding python instances
         self.columns = {}
-        self.columns_pp = {}
-                
-        i = 0
-        while i < view.column_count :
-            column_pp = <OBIDMS_column_p*> ((view.columns)+i)
-            column_p = <OBIDMS_column_p> (view.columns)[i]
-            header = (column_p).header
-            
-            col_name = bytes2str(header.name)
-            col_capsule = PyCapsule_New(column_pp, NULL, NULL)   # TODO discuss
-            (self.columns_pp)[col_name] = col_capsule
-            
-            subclass = OBIDMS_column.get_subclass_type(column_p)
+        for i in range(view.column_count) :
+            column_p               = <OBIDMS_column_p> (view.columns)[i]
+            header                 = (column_p).header
+            col_name               = bytes2str(header.name)         
+            subclass               = OBIDMS_column.get_subclass_type(column_p)
             self.columns[col_name] = subclass(self, col_name)
-            
-            i+=1
 
 
     def __repr__(self) :
         
         cdef str s
-        cdef OBIDMS_column column
-        cdef OBIDMS_column_p column_p
         
-        s = self.name
-        s = s + ", " + self.comments + ", " + str(self.pointer.line_count) + " lines" 
-        for column_name in self.columns :   # TODO make function in OBIDMS_column class
-            column = self.columns[column_name]
-            column_p = (column.pointer)[0]
-            s = s + "\n" + column_name + ", version " + str(column_p.header.version) + ", data type: " + column.data_type
+        s = str(self.name) + ", " + str(self.comments) + ", " + str(self.pointer.line_count) + " lines\n" 
+        for column_name in self.columns :
+            s = s + self.columns[column_name].__repr__() + '\n'
         return s
         
     
@@ -318,7 +311,6 @@ cdef class OBIView :
         cdef Obiview_p view
         cdef OBIDMS_column column
         cdef OBIDMS_column_p column_p
-        cdef OBIDMS_column_p* column_pp
         cdef OBIDMS_column_header_p header
         cdef str column_n
  
@@ -329,18 +321,7 @@ cdef class OBIView :
         
         # Update the dictionaries of column pointers and column objects, and update pointers in column objects (make function?):
         (self.columns).pop(column_name)
-        (self.columns_pp).pop(column_name)
         
-        i = 0
-        while i < view.column_count :
-            column_pp = <OBIDMS_column_p*> ((view.columns)+i)
-            column_p = <OBIDMS_column_p> (view.columns)[i]
-            header = (column_p).header
-            col_name = bytes2str(header.name)
-            col_capsule = PyCapsule_New(column_pp, NULL, NULL)
-            (self.columns_pp)[col_name] = col_capsule
-            i+=1
-
         for column_n in self.columns :
             (self.columns[column_n]).update_pointer()
 
@@ -350,12 +331,12 @@ cdef class OBIView :
                      obiversion_t version_number=-1, 
                      str type='',
                      index_t nb_lines=0, 
-                     index_t nb_elements_per_line=1,    # TODO 1?
+                     index_t nb_elements_per_line=1,
                      list elements_names=None,
-                     str avl_name="default_AVL_tree",
+                     str indexer_name="",
                      str comments="",
-                     bint create=True   # TODO
-                     ) :
+                     bint create=True
+                    ) :
         
         cdef bytes column_name_b
         cdef bytes elements_names_b
@@ -386,19 +367,14 @@ cdef class OBIView :
             else :
                 raise Exception("Invalid provided data type")
         
-        if (obi_view_add_column(self.pointer, column_name_b, version_number,     # should return pointer on column?
+        if (obi_view_add_column(self.pointer, column_name_b, version_number,     # TODO should return pointer on column?
                                 data_type, nb_lines, nb_elements_per_line, 
-                                elements_names_b, str2bytes(avl_name), 
+                                elements_names_b, str2bytes(indexer_name), 
                                 str2bytes(comments), create) < 0) :
             raise Exception("Problem adding a column in a view")
         
-        # Store the column pointer
+        # Get the column pointer
         column_pp = obi_view_get_pointer_on_column_in_view(self.pointer, column_name_b)
-        if column_pp == NULL :
-            raise Exception("Problem getting a column in a view")
-        
-        col_capsule = PyCapsule_New(column_pp, NULL, NULL)   # TODO
-        (self.columns_pp)[column_name] = col_capsule
         
         # Open and store the subclass
         column_p = column_pp[0]     # TODO ugly cython dereferencing
@@ -415,11 +391,11 @@ cdef class OBIView :
         # iter on each line of all columns
         
         # Declarations
-        cdef index_t lines_used
-        cdef index_t line_nb
-        cdef OBIView_line line  # TODO for NUC SEQS View
+        cdef index_t      lines_used
+        cdef index_t      line_nb
+        cdef OBIView_line line          # TODO Check that this works for NUC SEQ views
         
-        # Yield each line TODO line class
+        # Yield each line
         lines_used = (self.pointer).line_count
         
         for line_nb in range(lines_used) :
@@ -430,7 +406,7 @@ cdef class OBIView :
     def __getitem__(self, object item) :
         if type(item) == str :
             return (self.columns)[item]
-        elif type(item) == int :     # TODO int?
+        elif type(item) == int :
             return OBIView_line(self, item)
 
 
@@ -443,7 +419,7 @@ cdef class OBIView :
         cdef index_t* line_selection_p
         line_selection_p = <index_t*> malloc((len(line_selection) + 1) * sizeof(index_t))
         for i in range(len(line_selection)) :
-            line_selection_p[i] = line_selection[i]     # TODO type problem?
+            line_selection_p[i] = line_selection[i]
         line_selection_p[len(line_selection)] = -1
         if obi_select_lines(self.pointer, line_selection_p) < 0 :
             raise Exception("Problem selecting a list of lines")
@@ -467,22 +443,21 @@ cdef class OBIView_NUC_SEQS(OBIView):
     
     def __init__(self, OBIDMS dms, str view_name, bint new=False, object view_to_clone=None, list line_selection=None, str comments=""): 
         
-        cdef Obiview_p view = NULL
-        cdef int i
-        cdef list col_list
-        cdef str col_name
-        cdef OBIDMS_column column
-        cdef OBIDMS_column_p column_p
-        cdef OBIDMS_column_p* column_pp
+        cdef Obiview_p              view = NULL
+        cdef int                    i
+        cdef list                   col_list
+        cdef str                    col_name
+        cdef OBIDMS_column          column
+        cdef OBIDMS_column_p        column_p
         cdef OBIDMS_column_header_p header
-        cdef index_t* line_selection_p
+        cdef index_t*               line_selection_p
                 
         self.dms = dms
         
         if line_selection is not None :
             line_selection_p = <index_t*> malloc((len(line_selection) + 1) * sizeof(index_t))
             for i in range(len(line_selection)) :
-                line_selection_p[i] = line_selection[i]     # TODO type problem?
+                line_selection_p[i] = line_selection[i]
             line_selection_p[len(line_selection)] = -1
         else :
             line_selection_p = NULL
@@ -508,28 +483,18 @@ cdef class OBIView_NUC_SEQS(OBIView):
         self.name = bytes2str(view.name)
         self.comments = bytes2str(view.comments)
         
-        # go through columns to build list and open python object (TODO make separate function?)
+        # Go through columns to build list of corresponding python instances
         self.columns = {}
-        self.columns_pp = {}
-                
-        i = 0
-        while i < view.column_count :
-            column_pp = <OBIDMS_column_p*> ((view.columns)+i)
-            column_p = <OBIDMS_column_p> (view.columns)[i]
-            header = (column_p).header
-            
-            col_name = bytes2str(header.name)
-            col_capsule = PyCapsule_New(column_pp, NULL, NULL)   # TODO discuss
-            (self.columns_pp)[col_name] = col_capsule
-            
-            subclass = OBIDMS_column.get_subclass_type(column_p)
+        for i in range(view.column_count) :
+            column_p               = <OBIDMS_column_p> (view.columns)[i]
+            header                 = (column_p).header
+            col_name               = bytes2str(header.name)         
+            subclass               = OBIDMS_column.get_subclass_type(column_p)
             self.columns[col_name] = subclass(self, col_name)
-            
-            i+=1
         
         self.ids = self.columns[bytes2str(ID_COLUMN)]
         self.sequences = self.columns[bytes2str(NUC_SEQUENCE_COLUMN)]
-        self.descriptions = self.columns[bytes2str(DESCRIPTION_COLUMN)]
+        self.definitions = self.columns[bytes2str(DEFINITION_COLUMN)]
 
 
     cpdef delete_column(self, str column_name) :
@@ -538,11 +503,10 @@ cdef class OBIView_NUC_SEQS(OBIView):
         cdef Obiview_p view
         cdef OBIDMS_column column
         cdef OBIDMS_column_p column_p
-        cdef OBIDMS_column_p* column_pp
         cdef OBIDMS_column_header_p header
         cdef str column_n
  
-        if ((column_name == bytes2str(ID_COLUMN)) or (column_name == bytes2str(NUC_SEQUENCE_COLUMN)) or (column_name == bytes2str(DESCRIPTION_COLUMN))) :
+        if ((column_name == bytes2str(ID_COLUMN)) or (column_name == bytes2str(NUC_SEQUENCE_COLUMN)) or (column_name == bytes2str(DEFINITION_COLUMN))) :
             raise Exception("Can't delete an obligatory column from a NUC_SEQS view")
         
         view = self.pointer
@@ -550,20 +514,9 @@ cdef class OBIView_NUC_SEQS(OBIView):
         if obi_view_delete_column(view, str2bytes(column_name)) < 0 :
             raise Exception("Problem deleting a column from a view")
         
-        # Update the dictionaries of column pointers and column objects, and update pointers in column objects (make function?):
+        # Remove instance from the dictionary
         (self.columns).pop(column_name)
-        (self.columns_pp).pop(column_name)
         
-        i = 0
-        while i < view.column_count :
-            column_pp = <OBIDMS_column_p*> ((view.columns)+i)
-            column_p = <OBIDMS_column_p> (view.columns)[i]
-            header = (column_p).header
-            col_name = bytes2str(header.name)
-            col_capsule = PyCapsule_New(column_pp, NULL, NULL)
-            (self.columns_pp)[col_name] = col_capsule
-            i+=1
-
         for column_n in self.columns :
             (self.columns[column_n]).update_pointer()
 
@@ -571,7 +524,7 @@ cdef class OBIView_NUC_SEQS(OBIView):
     def __getitem__(self, object item) :
         if type(item) == str :
             return (self.columns)[item]
-        elif type(item) == int :     # TODO int?
+        elif type(item) == int :
             return OBI_Nuc_Seq_Stored(self, item)
     
     
@@ -606,7 +559,7 @@ cdef class OBIView_line :
             elif value_type == bool :
                 value_obitype = 'OBI_BOOL'            
             elif value_type == str :
-                if only_ATGC(str2bytes(value)) :    # TODO
+                if only_ATGC(str2bytes(value)) :    # TODO detect IUPAC?
                     value_obitype = 'OBI_SEQ'
                 elif len(value) == 1 :
                     value_obitype = 'OBI_CHAR'
@@ -690,20 +643,20 @@ cdef class OBIDMS :
         return all_views[view_name]
 
 
-    cpdef dict read_views(self) :    # TODO function that prints the dic nicely and function that prints 1 view. Add column type in col ref
+    cpdef dict read_views(self) :    # TODO function that prints the dic nicely and function that prints 1 view nicely. Add column type in col ref
         
         cdef Obiviews_infos_all_p all_views_p
-        cdef Obiview_infos_p view_p
-        cdef Column_reference_p column_refs
-        cdef int nb_views
-        cdef int i, j
-        cdef str view_name
-        cdef str column_name
-        cdef dict views
-        cdef bytes name_b
+        cdef Obiview_infos_p      view_p
+        cdef Column_reference_p   column_refs
+        cdef int                  nb_views
+        cdef int                  i, j
+        cdef str                  view_name
+        cdef str                  column_name
+        cdef dict                 views
+        cdef bytes                name_b
         
         views = {}
-        all_views_p = obi_read_views(self.pointer)
+        all_views_p = obi_read_view_infos(self.pointer)
         if all_views_p == NULL :
             raise Exception("No views to read")
         nb_views = <int> (all_views_p.header).view_count
@@ -717,7 +670,7 @@ cdef class OBIDMS :
             views[view_name]["line_count"] = <int> view_p.line_count
             views[view_name]["view_number"] = <int> view_p.view_number
             views[view_name]["created_from"] = bytes2str(view_p.created_from)
-            views[view_name]["creation_date"] = bytes2str(obi_column_format_date(view_p.creation_date))     # TODO move this function in utils or somethings
+            views[view_name]["creation_date"] = bytes2str(obi_format_date(view_p.creation_date))
             if (view_p.all_lines) :
                 views[view_name]["line_selection"] = None
             else :
@@ -731,7 +684,7 @@ cdef class OBIDMS :
                 views[view_name]["column_references"][column_name] = {}
                 views[view_name]["column_references"][column_name]["version"] = column_refs[j].version
         
-        obi_unmap_read_views(all_views_p);
+        obi_close_view_infos(all_views_p);
         
         return views
     

python/obitools3/obidms/_obidmscolumn_bool.cfiles

@@ -1,20 +1,51 @@
-../../../src/obidmscolumn_bool.c
-../../../src/obidmscolumn_bool.h
-../../../src/obiview.h
-../../../src/obiview.c
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
 ../../../src/obidmscolumndir.c
-../../../src/obidms.h
-../../../src/obidms.c
 ../../../src/obierrno.h
 ../../../src/obierrno.c
 ../../../src/obilittlebigman.h
 ../../../src/obilittlebigman.c
 ../../../src/obitypes.h
 ../../../src/obitypes.c
-../../../src/private_at_functions.h
-../../../src/private_at_functions.c
-../../../src/obiavl.h
-../../../src/obiavl.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/utils.h
+../../../src/utils.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h

python/obitools3/obidms/_obidmscolumn_bool.pyx

@@ -1,7 +1,6 @@
 #cython: language_level=3
  
-from .capi.obidmscolumn cimport obi_truncate_and_close_column, \
-                                obi_column_get_obibool_with_elt_name_in_view, \
+from .capi.obiview      cimport obi_column_get_obibool_with_elt_name_in_view, \
                                 obi_column_get_obibool_with_elt_idx_in_view, \
                                 obi_column_set_obibool_with_elt_name_in_view, \
                                 obi_column_set_obibool_with_elt_idx_in_view

python/obitools3/obidms/_obidmscolumn_char.cfiles

@@ -1,20 +1,51 @@
-../../../src/obidmscolumn_char.c
-../../../src/obidmscolumn_char.h
-../../../src/obiview.h
-../../../src/obiview.c
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
 ../../../src/obidmscolumndir.c
-../../../src/obidms.h
-../../../src/obidms.c
 ../../../src/obierrno.h
 ../../../src/obierrno.c
 ../../../src/obilittlebigman.h
 ../../../src/obilittlebigman.c
 ../../../src/obitypes.h
 ../../../src/obitypes.c
-../../../src/private_at_functions.h
-../../../src/private_at_functions.c
-../../../src/obiavl.h
-../../../src/obiavl.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/utils.h
+../../../src/utils.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h

python/obitools3/obidms/_obidmscolumn_char.pyx

@@ -1,7 +1,6 @@
 #cython: language_level=3
  
-from .capi.obidmscolumn cimport obi_truncate_and_close_column, \
-                                obi_column_get_obichar_with_elt_name_in_view, \
+from .capi.obiview      cimport obi_column_get_obichar_with_elt_name_in_view, \
                                 obi_column_get_obichar_with_elt_idx_in_view, \
                                 obi_column_set_obichar_with_elt_name_in_view, \
                                 obi_column_set_obichar_with_elt_idx_in_view

python/obitools3/obidms/_obidmscolumn_float.cfiles

@@ -1,20 +1,51 @@
-../../../src/obidmscolumn_float.c
-../../../src/obidmscolumn_float.h
-../../../src/obiview.h
-../../../src/obiview.c
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
 ../../../src/obidmscolumndir.c
-../../../src/obidms.h
-../../../src/obidms.c
 ../../../src/obierrno.h
 ../../../src/obierrno.c
 ../../../src/obilittlebigman.h
 ../../../src/obilittlebigman.c
 ../../../src/obitypes.h
 ../../../src/obitypes.c
-../../../src/private_at_functions.h
-../../../src/private_at_functions.c
-../../../src/obiavl.h
-../../../src/obiavl.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/utils.h
+../../../src/utils.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h

python/obitools3/obidms/_obidmscolumn_float.pxd

@@ -1,12 +1,13 @@
 #cython: language_level=3
 
-from .capi.obitypes cimport index_t
+from .capi.obitypes cimport index_t, obifloat_t
 from ._obidms       cimport OBIDMS_column, OBIDMS_column_multi_elts
 
 
 cdef class OBIDMS_column_float(OBIDMS_column):
     cpdef object get_line(self, index_t line_nb)
     cpdef set_line(self, index_t line_nb, object value)
+    cpdef list bool_list_from_compare(self, obifloat_t value, int comp)
 
 cdef class OBIDMS_column_multi_elts_float(OBIDMS_column_multi_elts):
     cpdef object get_item(self, index_t line_nb, str element_name)

python/obitools3/obidms/_obidmscolumn_float.pyx

@@ -1,12 +1,13 @@
 #cython: language_level=3
 
-from .capi.obidmscolumn cimport obi_truncate_and_close_column,\
-                                obi_column_get_obifloat_with_elt_name_in_view, \
+from .capi.obiview      cimport obi_column_get_obifloat_with_elt_name_in_view, \
                                 obi_column_get_obifloat_with_elt_idx_in_view, \
                                 obi_column_set_obifloat_with_elt_name_in_view, \
                                 obi_column_set_obifloat_with_elt_idx_in_view
 from .capi.obierrno     cimport obi_errno
-from .capi.obitypes     cimport OBIFloat_NA, obifloat_t
+from .capi.obitypes     cimport OBIFloat_NA, obifloat_t, byte_t
+from .capi.obidmscolumn cimport obi_float_column_greater_than   # TODO need to be in view...
+
 
 from obitools3.utils cimport str2bytes
 
@@ -31,6 +32,22 @@ cdef class OBIDMS_column_float(OBIDMS_column):
         if obi_column_set_obifloat_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0, <obifloat_t> value) < 0:
             raise Exception("Problem setting a value in a column")
 
+    cpdef list bool_list_from_compare(self, obifloat_t value, int comp):
+        cdef list    result_p
+        cdef byte_t* result_c
+        cdef int     i
+        result_p = []
+        if comp == 4 :
+            result_c = obi_float_get_bool_list_from_compare_in_view(self.view, (self.pointer)[0], value, GREATER_THAN)
+        else :
+            raise Exception("Comparison not handled")
+        for i in range(self.get_nb_lines_used()) :
+            result_p.append(<int> result_c[i])
+            print(result_p[i])
+        free(result_c)
+        return result_p
+
+
 
 cdef class OBIDMS_column_multi_elts_float(OBIDMS_column_multi_elts):
 

python/obitools3/obidms/_obidmscolumn_int.cfiles

@@ -1,20 +1,51 @@
-../../../src/obidmscolumn_int.c
-../../../src/obidmscolumn_int.h
-../../../src/obiview.h
-../../../src/obiview.c
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
 ../../../src/obidmscolumndir.c
-../../../src/obidms.h
-../../../src/obidms.c
 ../../../src/obierrno.h
 ../../../src/obierrno.c
 ../../../src/obilittlebigman.h
 ../../../src/obilittlebigman.c
 ../../../src/obitypes.h
 ../../../src/obitypes.c
-../../../src/private_at_functions.h
-../../../src/private_at_functions.c
-../../../src/obiavl.h
-../../../src/obiavl.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/utils.h
+../../../src/utils.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h

python/obitools3/obidms/_obidmscolumn_int.pyx

@@ -1,7 +1,6 @@
 #cython: language_level=3
 
-from .capi.obidmscolumn cimport obi_truncate_and_close_column, \
-                                obi_column_get_obiint_with_elt_name_in_view, \
+from .capi.obiview      cimport obi_column_get_obiint_with_elt_name_in_view, \
                                 obi_column_get_obiint_with_elt_idx_in_view, \
                                 obi_column_set_obiint_with_elt_name_in_view, \
                                 obi_column_set_obiint_with_elt_idx_in_view

python/obitools3/obidms/_obidmscolumn_seq.cfiles

@@ -1,20 +1,51 @@
-../../../src/obidmscolumn_seq.c
-../../../src/obidmscolumn_seq.h
-../../../src/obiview.h
-../../../src/obiview.c
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
 ../../../src/obidmscolumndir.c
-../../../src/obidms.h
-../../../src/obidms.c
 ../../../src/obierrno.h
 ../../../src/obierrno.c
 ../../../src/obilittlebigman.h
 ../../../src/obilittlebigman.c
 ../../../src/obitypes.h
 ../../../src/obitypes.c
-../../../src/private_at_functions.h
-../../../src/private_at_functions.c
-../../../src/obiavl.h
-../../../src/obiavl.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/utils.h
+../../../src/utils.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h

python/obitools3/obidms/_obidmscolumn_seq.pyx

@@ -1,7 +1,6 @@
 #cython: language_level=3
  
-from .capi.obidmscolumn cimport obi_truncate_and_close_column, \
-                                obi_column_get_obiseq_with_elt_name_in_view, \
+from .capi.obiview      cimport obi_column_get_obiseq_with_elt_name_in_view, \
                                 obi_column_get_obiseq_with_elt_idx_in_view, \
                                 obi_column_set_obiseq_with_elt_name_in_view, \
                                 obi_column_set_obiseq_with_elt_idx_in_view
@@ -10,19 +9,23 @@ from .capi.obitypes     cimport OBISeq_NA, const_char_p
 
 from obitools3.utils cimport str2bytes, bytes2str
 
+from libc.stdlib cimport free
+from libc.string cimport strcmp
+
 
 cdef class OBIDMS_column_seq(OBIDMS_column):
 
     cpdef object get_line(self, index_t line_nb):
-        cdef bytes value
+        cdef char*  value
         cdef object result
-        value = <bytes> obi_column_get_obiseq_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0)
+        value = obi_column_get_obiseq_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0)
         if obi_errno > 0 :
             raise IndexError(line_nb)
-        if value == OBISeq_NA :     # TODO
+        if strcmp(value, OBISeq_NA) == 0 :
             result = None
         else :
             result = bytes2str(value)
+            free(value)
         return result
 
     cpdef set_line(self, index_t line_nb, object value):
@@ -38,33 +41,35 @@ cdef class OBIDMS_column_seq(OBIDMS_column):
 cdef class OBIDMS_column_multi_elts_seq(OBIDMS_column_multi_elts):
 
     cpdef object get_item(self, index_t line_nb, str element_name):
-        cdef bytes value
+        cdef char*  value
         cdef object result
-        value = <bytes> obi_column_get_obiseq_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name))
+        value = obi_column_get_obiseq_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name))
         if obi_errno > 0 :
             raise IndexError(line_nb, element_name)
-        if value == OBISeq_NA :
+        if strcmp(value, OBISeq_NA) == 0 :
             result = None
         else :
             result = bytes2str(value)
+            free(value)
         return result
 
     cpdef object get_line(self, index_t line_nb) :
-        cdef bytes value
-        cdef object value_in_result
-        cdef dict result
+        cdef char*   value
+        cdef object  value_in_result
+        cdef dict    result
         cdef index_t i
-        cdef bint all_NA
+        cdef bint    all_NA
         result = {}
         all_NA = True
         for i in range(self.nb_elements_per_line) :
-            value = <bytes> obi_column_get_obiseq_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, i)
+            value = obi_column_get_obiseq_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, i)
             if obi_errno > 0 :
                 raise IndexError(line_nb)
-            if value == OBISeq_NA :
+            if strcmp(value, OBISeq_NA) == 0 :
                 value_in_result = None
             else :
-                value_in_result = bytes2str(value)    
+                value_in_result = bytes2str(value)
+                free(value)    
             result[self.elements_names[i]] = value_in_result
             if all_NA and (value_in_result is not None) :
                 all_NA = False

python/obitools3/obidms/_obidmscolumn_str.cfiles

@@ -1,20 +1,51 @@
-../../../src/obidmscolumn_str.c
-../../../src/obidmscolumn_str.h
-../../../src/obiview.h
-../../../src/obiview.c
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
 ../../../src/obidmscolumndir.c
-../../../src/obidms.h
-../../../src/obidms.c
 ../../../src/obierrno.h
 ../../../src/obierrno.c
 ../../../src/obilittlebigman.h
 ../../../src/obilittlebigman.c
 ../../../src/obitypes.h
 ../../../src/obitypes.c
-../../../src/private_at_functions.h
-../../../src/private_at_functions.c
-../../../src/obiavl.h
-../../../src/obiavl.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/utils.h
+../../../src/utils.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h

python/obitools3/obidms/_obidmscolumn_str.pyx

@@ -1,7 +1,6 @@
 #cython: language_level=3
  
-from .capi.obidmscolumn cimport obi_truncate_and_close_column, \
-                                obi_column_get_obistr_with_elt_name_in_view, \
+from .capi.obiview      cimport obi_column_get_obistr_with_elt_name_in_view, \
                                 obi_column_get_obistr_with_elt_idx_in_view, \
                                 obi_column_set_obistr_with_elt_name_in_view, \
                                 obi_column_set_obistr_with_elt_idx_in_view
@@ -10,19 +9,22 @@ from .capi.obitypes     cimport OBIStr_NA, const_char_p
 
 from obitools3.utils cimport str2bytes, bytes2str
 
+from libc.string cimport strcmp
+
 
 cdef class OBIDMS_column_str(OBIDMS_column):
 
     cpdef object get_line(self, index_t line_nb):
-        cdef bytes value
-        cdef object result
-        value = <bytes> obi_column_get_obistr_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0)
+        cdef const_char_p value
+        cdef object       result
+        value = obi_column_get_obistr_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0)
         if obi_errno > 0 :
             raise IndexError(line_nb)
-        if value == OBIStr_NA :     # TODO
+        if strcmp(value, OBIStr_NA) == 0 :
             result = None
         else :
             result = bytes2str(value)
+        # NOTE: value is not freed because the pointer points to a mmapped region in an AVL data file. (TODO discuss)
         return result
 
     cpdef set_line(self, index_t line_nb, object value):
@@ -38,33 +40,35 @@ cdef class OBIDMS_column_str(OBIDMS_column):
 cdef class OBIDMS_column_multi_elts_str(OBIDMS_column_multi_elts):
 
     cpdef object get_item(self, index_t line_nb, str element_name):
-        cdef bytes value
-        cdef object result
-        value = <bytes> obi_column_get_obistr_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name))
+        cdef const_char_p value
+        cdef object       result
+        value = obi_column_get_obistr_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name))
         if obi_errno > 0 :
             raise IndexError(line_nb, element_name)
-        if value == OBIStr_NA :
+        if strcmp(value, OBIStr_NA) == 0 :
             result = None
         else :
             result = bytes2str(value)
+        # NOTE: value is not freed because the pointer points to a mmapped region in an AVL data file. (TODO discuss)
         return result
 
     cpdef object get_line(self, index_t line_nb) :
-        cdef bytes value
-        cdef object value_in_result
-        cdef dict result
-        cdef index_t i
-        cdef bint all_NA
+        cdef const_char_p value
+        cdef object       value_in_result
+        cdef dict         result
+        cdef index_t      i
+        cdef bint         all_NA
         result = {}
         all_NA = True
         for i in range(self.nb_elements_per_line) :
-            value = <bytes> obi_column_get_obistr_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, i)
+            value = obi_column_get_obistr_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, i)
             if obi_errno > 0 :
                 raise IndexError(line_nb)
-            if value == OBIStr_NA :
+            if strcmp(value, OBIStr_NA) == 0 :
                 value_in_result = None
             else :
-                value_in_result = bytes2str(value)    
+                value_in_result = bytes2str(value)
+            # NOTE: value is not freed because the pointer points to a mmapped region in an AVL data file. (TODO discuss)
             result[self.elements_names[i]] = value_in_result
             if all_NA and (value_in_result is not None) :
                 all_NA = False

python/obitools3/obidms/_obiseq.cfiles

@@ -1,7 +1,28 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
 ../../../src/obidms.h
 ../../../src/obidms.c
-../../../src/obiview.h
-../../../src/obiview.c
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
@@ -12,11 +33,19 @@
 ../../../src/obilittlebigman.c
 ../../../src/obitypes.h
 ../../../src/obitypes.c
-../../../src/private_at_functions.h
-../../../src/private_at_functions.c
-../../../src/obiavl.h
-../../../src/obiavl.c
-../../../src/encode.h
-../../../src/encode.c
-../../../src/obidmscolumn_idx.h
-../../../src/obidmscolumn_idx.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/utils.h
+../../../src/utils.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h

python/obitools3/obidms/_obiseq.pxd

@@ -5,13 +5,13 @@ from ._obidms cimport OBIView_line
 
 cdef class OBI_Seq(dict) :
     cdef str id
-    cdef str description
+    cdef str definition
     cdef str sequence
 
     cpdef set_id(self, str id)
     cpdef get_id(self)
-    cpdef set_description(self, str description)
-    cpdef get_description(self)
+    cpdef set_definition(self, str definition)
+    cpdef get_definition(self)
     cpdef get_sequence(self)
 
 
@@ -21,14 +21,10 @@ cdef class OBI_Nuc_Seq(OBI_Seq) :
 
 
 cdef class OBI_Nuc_Seq_Stored(OBIView_line) :
-    cdef str id
-    cdef str description
-    cdef str sequence
-
     cpdef set_id(self, str id)
     cpdef get_id(self)
-    cpdef set_description(self, str description)
-    cpdef get_description(self)
+    cpdef set_definition(self, str definition)
+    cpdef get_definition(self)
     cpdef set_sequence(self, str sequence)
     cpdef get_sequence(self)
    # cpdef str reverse_complement(self)

python/obitools3/obidms/_obiseq.pyx

@@ -4,15 +4,15 @@ from obitools3.utils cimport bytes2str, str2bytes
 
 from .capi.obiview cimport NUC_SEQUENCE_COLUMN, \
                            ID_COLUMN, \
-                           DESCRIPTION_COLUMN
+                           DEFINITION_COLUMN
 
 cdef class OBI_Seq(dict) :
 
-    def __init__(self, str id, str seq, str description=None) :
+    def __init__(self, str id, str seq, str definition=None) :
         self.set_id(id)
         self.set_sequence(seq)
-        if description is not None :
-            self.set_description(description)
+        if definition is not None :
+            self.set_definition(definition)
         
     cpdef set_id(self, str id) :
         self.id = id
@@ -21,12 +21,12 @@ cdef class OBI_Seq(dict) :
     cpdef get_id(self) :
         return self.id
             
-    cpdef set_description(self, str description) :
-        self.description = description
-        self[bytes2str(DESCRIPTION_COLUMN)] = description
+    cpdef set_definition(self, str definition) :
+        self.definition = definition
+        self[bytes2str(DEFINITION_COLUMN)] = definition
         
-    cpdef get_description(self) :
-        return self.description
+    cpdef get_definition(self) :
+        return self.definition
         
     cpdef get_sequence(self) :
         return self.sequence
@@ -48,28 +48,25 @@ cdef class OBI_Nuc_Seq(OBI_Seq) :
 cdef class OBI_Nuc_Seq_Stored(OBIView_line) :
 
     cpdef set_id(self, str id) :
-        self.id = id
         self[bytes2str(ID_COLUMN)] = id
         
     cpdef get_id(self) :
-        return self.id
+        return self[bytes2str(ID_COLUMN)]
             
-    cpdef set_description(self, str description) :
-        self.description = description
-        self[bytes2str(DESCRIPTION_COLUMN)] = description
+    cpdef set_definition(self, str definition) :
+        self[bytes2str(DEFINITION_COLUMN)] = definition
 
-    cpdef get_description(self) :
-        return self.description
+    cpdef get_definition(self) :
+        return self[bytes2str(DEFINITION_COLUMN)]
 
     cpdef set_sequence(self, str sequence) :
-        self.sequence = sequence
         self[bytes2str(NUC_SEQUENCE_COLUMN)] = sequence
 
     cpdef get_sequence(self) :
-        return self.sequence
+        return self[bytes2str(NUC_SEQUENCE_COLUMN)]
     
-    def __str__(self) :
-        return self.sequence    # or not
+#    def __str__(self) :
+#        return self[bytes2str(NUC_SEQUENCE_COLUMN)]    # or not
     
 #    cpdef str reverse_complement(self) :    TODO in C ?
 #        pass

python/obitools3/obidms/_obitaxo.cfiles

@@ -1,7 +1,28 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
 ../../../src/obidms.h
 ../../../src/obidms.c
-../../../src/obiview.h
-../../../src/obiview.c
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
@@ -12,13 +33,19 @@
 ../../../src/obilittlebigman.c
 ../../../src/obitypes.h
 ../../../src/obitypes.c
-../../../src/private_at_functions.h
-../../../src/private_at_functions.c
-../../../src/obiavl.h
-../../../src/obiavl.c
-../../../src/encode.h
-../../../src/encode.c
-../../../src/obidmscolumn_idx.h
-../../../src/obidmscolumn_idx.c
-../../../src/obidms_taxonomy.c
-../../../src/obidms_taxonomy.h
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/utils.h
+../../../src/utils.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h

python/obitools3/obidms/capi/obidmscolumn.pxd

@@ -27,7 +27,7 @@ cdef extern from "obidmscolumn.h" nogil:
         obiversion_t    version
         obiversion_t    cloned_from
         const_char_p    name
-        const_char_p    avl_name
+        const_char_p    indexer_name
         const_char_p    comments
     
     ctypedef OBIDMS_column_header_t* OBIDMS_column_header_p
@@ -45,7 +45,7 @@ cdef extern from "obidmscolumn.h" nogil:
                                       index_t nb_lines, 
                                       index_t nb_elements_per_line, 
                                       const_char_p elements_names,
-                                      const_char_p avl_name,
+                                      const_char_p indexer_name,
                                       const_char_p comments)
                 
     OBIDMS_column_p obi_open_column(OBIDMS_p dms, 
@@ -60,7 +60,7 @@ cdef extern from "obidmscolumn.h" nogil:
                                      obiversion_t version_number, 
                                      bint clone_data)
     
-    int obi_truncate_and_close_column(OBIDMS_column_p column)
+    int obi_close_column(OBIDMS_column_p column)
     
     obiversion_t obi_column_get_latest_version_from_name(OBIDMS_p dms, 
                                                          const_char_p column_name)
@@ -69,37 +69,12 @@ cdef extern from "obidmscolumn.h" nogil:
                                                            const_char_p column_name,
                                                            obiversion_t version_number)
     
-    int obi_unmap_header(OBIDMS_column_header_p header)
-        
-    char* obi_column_format_date(time_t date)
+    int obi_close_header(OBIDMS_column_header_p header)
 
     int obi_select(OBIDMS_column_p line_selection_column, index_t line_to_grep)
 
 
-from ..capi.obiview  cimport Obiview_p  # TODO
-
 cdef extern from "obidmscolumn_int.h" nogil:
-    int obi_column_set_obiint_with_elt_name_in_view(Obiview_p view,
-                                                    OBIDMS_column_p column, 
-                                                    index_t line_nb, 
-                                                    const_char_p element_name, 
-                                                    obiint_t value)
-
-    int obi_column_set_obiint_with_elt_idx_in_view(Obiview_p view,
-                                                   OBIDMS_column_p column, 
-                                                   index_t line_nb, 
-                                                   index_t element_idx, 
-                                                   obiint_t value)
-    
-    obiint_t obi_column_get_obiint_with_elt_name_in_view(Obiview_p view,
-                                                         OBIDMS_column_p column, 
-                                                         index_t line_nb, 
-                                                         const_char_p element_name)
-    
-    obiint_t obi_column_get_obiint_with_elt_idx_in_view(Obiview_p view,
-                                                        OBIDMS_column_p column, 
-                                                        index_t line_nb, 
-                                                        index_t element_idx)
 
     int obi_column_set_obiint_with_elt_name(OBIDMS_column_p column, 
                                             index_t line_nb, 
@@ -121,28 +96,6 @@ cdef extern from "obidmscolumn_int.h" nogil:
 
 cdef extern from "obidmscolumn_bool.h" nogil:
 
-    int obi_column_set_obibool_with_elt_name_in_view(Obiview_p view,
-                                                     OBIDMS_column_p column, 
-                                                     index_t line_nb, 
-                                                     const_char_p element_name, 
-                                                     obibool_t value)
-
-    int obi_column_set_obibool_with_elt_idx_in_view(Obiview_p view,
-                                                    OBIDMS_column_p column, 
-                                                    index_t line_nb, 
-                                                    index_t element_idx, 
-                                                    obibool_t value)
-    
-    obibool_t obi_column_get_obibool_with_elt_name_in_view(Obiview_p view,
-                                                          OBIDMS_column_p column, 
-                                                          index_t line_nb, 
-                                                          const_char_p element_name)
-    
-    obibool_t obi_column_get_obibool_with_elt_idx_in_view(Obiview_p view,
-                                                         OBIDMS_column_p column, 
-                                                         index_t line_nb, 
-                                                         index_t element_idx)
-
     int obi_column_set_obibool_with_elt_name(OBIDMS_column_p column, 
                                              index_t line_nb, 
                                              const_char_p element_name, 
@@ -163,28 +116,6 @@ cdef extern from "obidmscolumn_bool.h" nogil:
 
 cdef extern from "obidmscolumn_char.h" nogil:
 
-    int obi_column_set_obichar_with_elt_name_in_view(Obiview_p view,
-                                                     OBIDMS_column_p column, 
-                                                     index_t line_nb, 
-                                                     const_char_p element_name, 
-                                                     obichar_t value)
-
-    int obi_column_set_obichar_with_elt_idx_in_view(Obiview_p view,
-                                                    OBIDMS_column_p column, 
-                                                    index_t line_nb, 
-                                                    index_t element_idx, 
-                                                    obichar_t value)
-    
-    obichar_t obi_column_get_obichar_with_elt_name_in_view(Obiview_p view,
-                                                          OBIDMS_column_p column, 
-                                                          index_t line_nb, 
-                                                          const_char_p element_name)
-    
-    obichar_t obi_column_get_obichar_with_elt_idx_in_view(Obiview_p view,
-                                                         OBIDMS_column_p column, 
-                                                         index_t line_nb, 
-                                                         index_t element_idx)
-
     int obi_column_set_obichar_with_elt_name(OBIDMS_column_p column, 
                                              index_t line_nb, 
                                              const_char_p element_name, 
@@ -205,83 +136,39 @@ cdef extern from "obidmscolumn_char.h" nogil:
 
 cdef extern from "obidmscolumn_float.h" nogil:
 
-    int obi_column_set_obifloat_with_elt_name_in_view(Obiview_p view,
-                                                     OBIDMS_column_p column, 
-                                                     index_t line_nb, 
-                                                     const_char_p element_name, 
-                                                     obifloat_t value)
-
-    int obi_column_set_obifloat_with_elt_idx_in_view(Obiview_p view,
-                                                    OBIDMS_column_p column, 
-                                                    index_t line_nb, 
-                                                    index_t element_idx, 
-                                                    obifloat_t value)
-    
-    obifloat_t obi_column_get_obifloat_with_elt_name_in_view(Obiview_p view,
-                                                          OBIDMS_column_p column, 
-                                                          index_t line_nb, 
-                                                          const_char_p element_name)
-    
-    obifloat_t obi_column_get_obifloat_with_elt_idx_in_view(Obiview_p view,
-                                                         OBIDMS_column_p column, 
-                                                         index_t line_nb, 
-                                                         index_t element_idx)
-
     int obi_column_set_obifloat_with_elt_name(OBIDMS_column_p column, 
-                                             index_t line_nb, 
-                                             const_char_p element_name, 
-                                             obifloat_t value)
+                                              index_t line_nb, 
+                                              const_char_p element_name, 
+                                              obifloat_t value)
 
     int obi_column_set_obifloat_with_elt_idx(OBIDMS_column_p column, 
-                                            index_t line_nb, 
-                                            index_t element_idx, 
-                                            obifloat_t value)
+                                             index_t line_nb, 
+                                             index_t element_idx, 
+                                             obifloat_t value)
     
     obifloat_t obi_column_get_obifloat_with_elt_name(OBIDMS_column_p column, 
-                                                   index_t line_nb, 
-                                                   const_char_p element_name)
+                                                     index_t line_nb, 
+                                                     const_char_p element_name)
     
     obifloat_t obi_column_get_obifloat_with_elt_idx(OBIDMS_column_p column, 
-                                                  index_t line_nb, 
-                                                  index_t element_idx)
+                                                    index_t line_nb, 
+                                                    index_t element_idx)
 
 cdef extern from "obidmscolumn_str.h" nogil:
 
-    int obi_column_set_obistr_with_elt_name_in_view(Obiview_p view,
-                                                    OBIDMS_column_p column, 
-                                                    index_t line_nb, 
-                                                    const_char_p element_name, 
-                                                    char* value)
-
-    int obi_column_set_obistr_with_elt_idx_in_view(Obiview_p view,
-                                                   OBIDMS_column_p column, 
-                                                   index_t line_nb, 
-                                                   index_t element_idx, 
-                                                   char* value)
-    
-    const_char_p obi_column_get_obistr_with_elt_name_in_view(Obiview_p view,
-                                                           OBIDMS_column_p column, 
-                                                           index_t line_nb, 
-                                                           const_char_p element_name)
-    
-    const_char_p obi_column_get_obistr_with_elt_idx_in_view(Obiview_p view,
-                                                            OBIDMS_column_p column, 
-                                                            index_t line_nb, 
-                                                            index_t element_idx)
-
     int obi_column_set_obistr_with_elt_name(OBIDMS_column_p column, 
                                             index_t line_nb, 
                                             const_char_p element_name, 
-                                            char* value)
+                                            const_char_p value)
 
     int obi_column_set_obistr_with_elt_idx(OBIDMS_column_p column, 
                                            index_t line_nb, 
                                            index_t element_idx, 
-                                           char* value)
+                                           const_char_p value)
     
     const_char_p obi_column_get_obistr_with_elt_name(OBIDMS_column_p column, 
-                                                    index_t line_nb, 
-                                                    char* element_name)
+                                                     index_t line_nb, 
+                                                     const_char_p element_name)
     
     const_char_p obi_column_get_obistr_with_elt_idx(OBIDMS_column_p column, 
                                                     index_t line_nb, 
@@ -289,43 +176,21 @@ cdef extern from "obidmscolumn_str.h" nogil:
 
 cdef extern from "obidmscolumn_seq.h" nogil:
 
-    int obi_column_set_obiseq_with_elt_name_in_view(Obiview_p view,
-                                                    OBIDMS_column_p column, 
-                                                    index_t line_nb, 
-                                                    const_char_p element_name, 
-                                                    char* value)
-
-    int obi_column_set_obiseq_with_elt_idx_in_view(Obiview_p view,
-                                                   OBIDMS_column_p column, 
-                                                   index_t line_nb, 
-                                                   index_t element_idx, 
-                                                   char* value)
-    
-    const_char_p obi_column_get_obiseq_with_elt_name_in_view(Obiview_p view,
-                                                           OBIDMS_column_p column, 
-                                                           index_t line_nb, 
-                                                           const_char_p element_name)
-    
-    const_char_p obi_column_get_obiseq_with_elt_idx_in_view(Obiview_p view,
-                                                            OBIDMS_column_p column, 
-                                                            index_t line_nb, 
-                                                            index_t element_idx)
-
     int obi_column_set_obiseq_with_elt_name(OBIDMS_column_p column, 
                                             index_t line_nb, 
                                             const_char_p element_name, 
-                                            char* value)
+                                            const_char_p value)
 
     int obi_column_set_obiseq_with_elt_idx(OBIDMS_column_p column, 
                                            index_t line_nb, 
                                            index_t element_idx, 
-                                           char* value)
+                                           const_char_p value)
     
-    const_char_p obi_column_get_obiseq_with_elt_name(OBIDMS_column_p column, 
-                                                    index_t line_nb, 
-                                                    char* element_name)
+    char* obi_column_get_obiseq_with_elt_name(OBIDMS_column_p column, 
+                                              index_t line_nb, 
+                                              const_char_p element_name)
     
-    const_char_p obi_column_get_obiseq_with_elt_idx(OBIDMS_column_p column, 
-                                                    index_t line_nb, 
-                                                    index_t element_idx)
+    char* obi_column_get_obiseq_with_elt_idx(OBIDMS_column_p column, 
+                                             index_t line_nb, 
+                                             index_t element_idx)
 

python/obitools3/obidms/capi/obitypes.pxd

@@ -8,15 +8,11 @@ from posix.types cimport time_t
 cdef extern from *:
     ctypedef char* const_char_p "const char*"
 
+
 cdef extern from "encode.h" nogil:
     bint only_ATGC(const_char_p seq)
 
 
-cdef extern from "obidmscolumn.h" nogil:
-
-    ctypedef int32_t obiversion_t
-
-
 cdef extern from "obitypes.h" nogil:
 
     enum OBIType:
@@ -37,10 +33,12 @@ cdef extern from "obitypes.h" nogil:
     
     ctypedef OBIBool obibool_t
     ctypedef int32_t obiint_t
-    ctypedef double obifloat_t
-    ctypedef char obichar_t  
+    ctypedef double  obifloat_t
+    ctypedef char    obichar_t  
     ctypedef int64_t index_t
 
+    ctypedef int32_t obiversion_t
+
     extern obiint_t      OBIInt_NA
     extern index_t       OBIIdx_NA
     extern obifloat_t    OBIFloat_NA

python/obitools3/obidms/capi/obiutils.pxd

@@ -0,0 +1,12 @@
+#cython: language_level=3
+
+
+from posix.types cimport time_t
+
+from ..capi.obitypes cimport const_char_p
+
+
+cdef extern from "utils.h" nogil:
+
+    const_char_p obi_format_date(time_t date)
+

python/obitools3/obidms/capi/obiview.pxd

@@ -3,6 +3,10 @@
 from .obitypes           cimport const_char_p, \
                                  OBIType_t, \
                                  obiversion_t, \
+                                 obiint_t, \
+                                 obibool_t, \
+                                 obichar_t, \
+                                 obifloat_t, \
                                  index_t, \
                                  time_t
 from ..capi.obidms       cimport OBIDMS_p
@@ -14,7 +18,23 @@ cdef extern from "obiview.h" nogil:
     extern const_char_p VIEW_TYPE_NUC_SEQS
     extern const_char_p NUC_SEQUENCE_COLUMN
     extern const_char_p ID_COLUMN
-    extern const_char_p DESCRIPTION_COLUMN
+    extern const_char_p DEFINITION_COLUMN
+
+    struct Obiview_t :
+        OBIDMS_p            dms
+        const_char_p        name
+        const_char_p        created_from
+        const_char_p        view_type
+        bint                read_only
+        OBIDMS_column_p     line_selection
+        OBIDMS_column_p     new_line_selection
+        index_t             line_count
+        int                 column_count
+        OBIDMS_column_p     columns
+        const_char_p        comments
+
+    ctypedef Obiview_t* Obiview_p
+
 
     struct Column_reference_t :
         const_char_p  column_name
@@ -23,32 +43,17 @@ cdef extern from "obiview.h" nogil:
     ctypedef Column_reference_t* Column_reference_p
 
 
-    struct Obiview_t :
-        OBIDMS_p            dms
-        const_char_p        name
-        OBIDMS_column_p     line_selection
-        OBIDMS_column_p     new_line_selection
-        OBIDMS_column_p     columns
-        bint                read_only
-        Column_reference_t  line_selection_reference
-        index_t             line_count
-        int                 column_count
-        const_char_p        comments
-
-    ctypedef Obiview_t* Obiview_p
-
-
     struct Obiview_infos_t :
         int                 view_number
-        int                 column_count
-        index_t             line_count
+        time_t              creation_date
         const_char_p        name
         const_char_p        created_from
-        time_t              creation_date
+        const_char_p        view_type
         bint                all_lines
         Column_reference_t  line_selection
+        index_t             line_count
+        int                 column_count
         Column_reference_p  column_references
-        const_char_p        view_type
         const_char_p        comments
 
     ctypedef Obiview_infos_t* Obiview_infos_p
@@ -79,10 +84,6 @@ cdef extern from "obiview.h" nogil:
               
     Obiview_p obi_open_view(OBIDMS_p dms, const_char_p view_name)
     
-    Obiviews_infos_all_p obi_read_views(OBIDMS_p dms)
-
-    int obi_unmap_read_views(Obiviews_infos_all_p views)
-
     int obi_view_add_column(Obiview_p view,
                             const_char_p column_name,
                             obiversion_t version_number,
@@ -90,7 +91,7 @@ cdef extern from "obiview.h" nogil:
                             index_t nb_lines,
                             index_t nb_elements_per_line,
                             const_char_p elements_names,
-                            const_char_p avl_name,
+                            const_char_p indexer_name,
                             const_char_p comments,
                             bint create)
 
@@ -99,8 +100,6 @@ cdef extern from "obiview.h" nogil:
     int obi_select_line(Obiview_p view, index_t line_nb)
 
     int obi_select_lines(Obiview_p view, index_t* line_nbs)
-
-    OBIDMS_column_p obi_view_clone_column(Obiview_p view, const_char_p column_name)
         
     OBIDMS_column_p obi_view_get_column(Obiview_p view, const_char_p column_name)
 
@@ -111,4 +110,140 @@ cdef extern from "obiview.h" nogil:
     int obi_close_view(Obiview_p view)
     
     int obi_save_and_close_view(Obiview_p view)
+    
+    Obiviews_infos_all_p obi_read_view_infos(OBIDMS_p dms)
+
+    int obi_close_view_infos(Obiviews_infos_all_p views)
+
+    int obi_column_set_obiint_with_elt_name_in_view(Obiview_p view,
+                                                    OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    const_char_p element_name, 
+                                                    obiint_t value)
+
+    int obi_column_set_obiint_with_elt_idx_in_view(Obiview_p view,
+                                                   OBIDMS_column_p column, 
+                                                   index_t line_nb, 
+                                                   index_t element_idx, 
+                                                   obiint_t value)
+    
+    obiint_t obi_column_get_obiint_with_elt_name_in_view(Obiview_p view,
+                                                         OBIDMS_column_p column, 
+                                                         index_t line_nb, 
+                                                         const_char_p element_name)
+    
+    obiint_t obi_column_get_obiint_with_elt_idx_in_view(Obiview_p view,
+                                                        OBIDMS_column_p column, 
+                                                        index_t line_nb, 
+                                                        index_t element_idx)
+
+    int obi_column_set_obibool_with_elt_name_in_view(Obiview_p view,
+                                                     OBIDMS_column_p column, 
+                                                     index_t line_nb, 
+                                                     const_char_p element_name, 
+                                                     obibool_t value)
+
+    int obi_column_set_obibool_with_elt_idx_in_view(Obiview_p view,
+                                                    OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    index_t element_idx, 
+                                                    obibool_t value)
+    
+    obibool_t obi_column_get_obibool_with_elt_name_in_view(Obiview_p view,
+                                                           OBIDMS_column_p column, 
+                                                           index_t line_nb, 
+                                                           const_char_p element_name)
+    
+    obibool_t obi_column_get_obibool_with_elt_idx_in_view(Obiview_p view,
+                                                          OBIDMS_column_p column, 
+                                                          index_t line_nb, 
+                                                          index_t element_idx)
+
+    int obi_column_set_obichar_with_elt_name_in_view(Obiview_p view,
+                                                     OBIDMS_column_p column, 
+                                                     index_t line_nb, 
+                                                     const_char_p element_name, 
+                                                     obichar_t value)
+
+    int obi_column_set_obichar_with_elt_idx_in_view(Obiview_p view,
+                                                    OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    index_t element_idx, 
+                                                    obichar_t value)
+    
+    obichar_t obi_column_get_obichar_with_elt_name_in_view(Obiview_p view,
+                                                           OBIDMS_column_p column, 
+                                                           index_t line_nb, 
+                                                           const_char_p element_name)
+    
+    obichar_t obi_column_get_obichar_with_elt_idx_in_view(Obiview_p view,
+                                                          OBIDMS_column_p column, 
+                                                          index_t line_nb, 
+                                                          index_t element_idx)
+
+    int obi_column_set_obifloat_with_elt_name_in_view(Obiview_p view,
+                                                      OBIDMS_column_p column, 
+                                                      index_t line_nb, 
+                                                      const_char_p element_name, 
+                                                      obifloat_t value)
+
+    int obi_column_set_obifloat_with_elt_idx_in_view(Obiview_p view,
+                                                     OBIDMS_column_p column, 
+                                                     index_t line_nb, 
+                                                     index_t element_idx, 
+                                                     obifloat_t value)
+    
+    obifloat_t obi_column_get_obifloat_with_elt_name_in_view(Obiview_p view,
+                                                             OBIDMS_column_p column, 
+                                                             index_t line_nb, 
+                                                             const_char_p element_name)
+    
+    obifloat_t obi_column_get_obifloat_with_elt_idx_in_view(Obiview_p view,
+                                                            OBIDMS_column_p column, 
+                                                            index_t line_nb, 
+                                                            index_t element_idx)
+
+    int obi_column_set_obistr_with_elt_name_in_view(Obiview_p view,
+                                                    OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    const_char_p element_name, 
+                                                    const_char_p value)
+
+    int obi_column_set_obistr_with_elt_idx_in_view(Obiview_p view,
+                                                   OBIDMS_column_p column, 
+                                                   index_t line_nb, 
+                                                   index_t element_idx, 
+                                                   const_char_p value)
+    
+    const_char_p obi_column_get_obistr_with_elt_name_in_view(Obiview_p view,
+                                                           OBIDMS_column_p column, 
+                                                           index_t line_nb, 
+                                                           const_char_p element_name)
+    
+    const_char_p obi_column_get_obistr_with_elt_idx_in_view(Obiview_p view,
+                                                            OBIDMS_column_p column, 
+                                                            index_t line_nb, 
+                                                            index_t element_idx)
+
+    int obi_column_set_obiseq_with_elt_name_in_view(Obiview_p view,
+                                                    OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    const_char_p element_name, 
+                                                    const_char_p value)
+
+    int obi_column_set_obiseq_with_elt_idx_in_view(Obiview_p view,
+                                                   OBIDMS_column_p column, 
+                                                   index_t line_nb, 
+                                                   index_t element_idx, 
+                                                   const_char_p value)
+    
+    char* obi_column_get_obiseq_with_elt_name_in_view(Obiview_p view,
+                                                      OBIDMS_column_p column, 
+                                                      index_t line_nb, 
+                                                      const_char_p element_name)
+    
+    char* obi_column_get_obiseq_with_elt_idx_in_view(Obiview_p view,
+                                                     OBIDMS_column_p column, 
+                                                     index_t line_nb, 
+                                                     index_t element_idx)
 

python/obitools3/obigrep.py

@@ -0,0 +1,51 @@
+import sys
+import argparse
+
+from obitools3.obidms._obidms import OBIDMS
+
+if __name__ == '__main__': 
+    
+    parser = argparse.ArgumentParser(description='Pseudo obigrep.')
+    
+    parser.add_argument('-V', '--view', dest='view', type=str,
+                       help='Name of the view that should be considered')    
+
+    parser.add_argument('-N', '--new_view', dest='new_view', type=str,
+                       help='Name of the new view that should be created')    
+    
+#     parser.add_argument('-k', '--key', dest='key', type=str,
+#                        help='Name of the key that should be considered')
+# 
+#     parser.add_argument('-c', '--comp', dest='comparison', type=int,
+#                        help='Comparison to be made: -1:< ; 0:== ; 1:>')
+# 
+#     parser.add_argument('-v', '--value', dest='value', type=object,
+#                        help='Value to be compared')
+
+    args = parser.parse_args()
+    
+    d = OBIDMS('tdms')
+    
+    #condition = 1
+    line_selec = []
+    
+    v = d.open_view(args.view)
+    
+    i = 0
+    for l in v :
+        if l['score'] > 350 : 
+            line_selec.append(i)
+        i+=1
+        
+    new_v = d.new_view(args.new_view, view_to_clone=v, line_selection=line_selec, view_type="NUC_SEQS_VIEW", comments="obigrep "+args.view+" to "+args.new_view) #args.key+" "+str(args.comparison)+" "+str(args.value)+" "+)
+
+    print("\n")
+    print(new_v.__repr__())
+
+    v.save_and_close()
+    new_v.save_and_close()
+    
+    d.close()
+
+    print("\nDone.")
+    

python/obitools3/obihead.py

@@ -0,0 +1,43 @@
+import sys
+import argparse
+
+from obitools3.obidms._obidms import OBIDMS
+
+if __name__ == '__main__': 
+    
+    parser = argparse.ArgumentParser(description='Pseudo obihead.')
+    
+    parser.add_argument('-V', '--view', dest='view', type=str,
+                       help='Name of the view that should be considered')    
+
+    parser.add_argument('-N', '--new_view', dest='new_view', type=str,
+                       help='Name of the new view that should be created')    
+    
+    parser.add_argument('-n', '--nb', dest='nb_lines', type=int,
+                       help='Number of lines that should be taken')    
+    
+
+    args = parser.parse_args()
+    
+    d = OBIDMS('tdms')
+    
+    #condition = 1
+    line_selec = []
+    
+    v = d.open_view(args.view)
+    
+    for i in range(0, args.nb_lines) :
+        line_selec.append(i)
+        
+    new_v = d.new_view(args.new_view, view_to_clone=v, line_selection=line_selec, view_type="NUC_SEQS_VIEW", comments="obihead "+str(args.nb_lines)+", "+args.view+" to "+args.new_view) #args.key+" "+str(args.comparison)+" "+str(args.value)+" "+)
+
+    print("\n")
+    print(new_v.__repr__())
+
+    v.save_and_close()
+    new_v.save_and_close()
+    
+    d.close()
+
+    print("\nDone.")
+    

python/obitools3/obiimport.py

@@ -0,0 +1,199 @@
+import sys
+import argparse
+import time
+
+from obitools3.obidms._obidms import OBIDMS
+
+
+def bufferedRead(fileobj,size=209715200):   ## 200 MB
+   buffer = fileobj.readlines(size)
+   while buffer:
+       for l in buffer:
+           yield l
+       buffer = fileobj.readlines(size)
+
+
+if __name__ == '__main__': 
+    
+    parser = argparse.ArgumentParser(description='Convert a fasta file in an OBIDMS.')
+    
+    parser.add_argument('-i', '--input', dest='input_file', type=str,
+                       help='Name of the file containing the sequences')
+
+    args = parser.parse_args()
+
+    d = OBIDMS('tdms')
+    
+    view = d.new_view('uniq view', view_type="NUC_SEQS_VIEW")
+
+#    for i in range(35000000) :
+#        if (not (i%500000)) :
+#            print(str(time.time())+'\t'+str(i))
+#        id = "@HWI-D00405:142:C71BAANXX:4:1101:1234:2234_CONS_SUB_SUB_"+str(i)
+#        view[i].set_id(id)
+#        if id != view[i]["ID"] :
+#            print("nope", id, view[i]["ID"])
+    
+    input_file = open(args.input_file, 'r')
+    input_file_buffered = bufferedRead(input_file)
+    
+#         
+#     if args.input_file[-1:] == "a" :
+#     
+#         i = 0
+#         next = False
+#         first = True
+#         
+#         for line in input_file :
+#             
+#             if line[0] == ">" :
+#                 
+#                 if not first :
+#                     # save seq
+#                     #print(i, id, seq)
+#                     view[i].set_sequence(seq)
+#                     i+=1
+#                 
+#                 first = False
+#                     
+#                 #id = line.split(" ", 1)[0][1:]
+#                 #rest = (line[:-1].split(" ", 1)[1]).split(";")
+#                 #view[i].set_id(id)
+#                 
+# #                 description = ""
+# #                 for j in range(len(rest)) :
+# #                     if "=" in rest[j] :
+# #                         rest[j] = rest[j].strip()
+# #                         rest[j] = rest[j].split("=", 1)
+# #                         column_name = rest[j][0]
+# #                         v = rest[j][1]
+# #                         if ((not v.isalpha()) and (v.isalnum())) :
+# #                             conv_v = int(v)
+# #                         elif (v == "True") or (v == "False") :
+# #                             conv_v = bool(v)
+# #                         else :
+# #                             f = True
+# #                             for letter in v :
+# #                                 if ((not letter.isalnum()) or (letter != ".")) :
+# #                                     f = False
+# #                             if f :
+# #                                 conv_v = float(v)
+# #                             else : 
+# #                                 conv_v = v
+# #                         view[i][column_name] = conv_v
+# #                     else :
+# #                         description+=rest[j]
+# #                 
+# #                 if description != "" :
+# #                     description = description.strip()
+# #                     view[i].set_description(description)
+#                     
+#                 #print(id)
+#                 #print(rest)
+#                 #print(description)
+#                 
+#                 next = True
+#                 
+#             elif next == True :
+#                 
+#      #           if not (i % 1E5) :
+#      #               print(i)
+#                 
+#                 seq = line[:-1]
+#                 next = False
+#             
+#             elif not next :
+#                 
+#                 seq += line[:-1]
+# 
+# 
+#     elif args.input_file[-1:] == "q" :
+#         
+#         i = 0
+#         l = 0
+#         next = False
+#
+    l=0
+    i=0
+#     while (True):
+#         l+=1
+#         line = input_file.readline()
+#         if line=="":
+#             break
+    for line in input_file_buffered :
+# 
+             #if i > 1E7 :
+#             #    print('hmm?')
+#                         
+#        if i == 6000000 :
+#            break
+#             
+        if l%4 == 0 :
+#                
+            if (not (i%500000)) :
+                print(str(time.time())+'\t'+str(i))
+# # 
+# #                 #print("header", line)
+# # 
+            id = line.split(" ", 1)[0][1:]
+#            print(id)
+# #                 #rest = (line[:-1].split(" ", 1)[1]).split(";")
+            view[i].set_id(id)
+#            print(view[i]["ID"])
+# 
+#            i+=1
+        
+#        l+=1
+#                 
+# #                 description = ""
+# #                 for j in range(len(rest)) :
+# #                     if "=" in rest[j] :
+# #                         rest[j] = rest[j].strip()
+# #                         rest[j] = rest[j].split("=", 1)
+# #                         column_name = rest[j][0]
+# #                         #print("COLUMN", column_name)
+# #                         v = rest[j][1]
+# #                         if (v == "") and (column_name in view) and (view[column_name].get_data_type() == "OBI_SEQ") :
+# #                             #print(">>>>>>YUP")
+# #                             conv_v = "aa"
+# #                         else :
+# #                             if ((not v.isalpha()) and (v.isalnum())) :
+# #                                 conv_v = int(v)
+# #                             elif (v == "True") or (v == "False") :
+# #                                 conv_v = bool(v)
+# #                             else :
+# #                                 f = True
+# #                                 for letter in v :
+# #                                     if ((not letter.isalnum()) or (letter != ".")) :
+# #                                         f = False
+# #                                 if f :
+# #                                     conv_v = float(v)
+# #                                 else :
+# #                                     conv_v = v
+# #                         view[i][column_name] = conv_v
+# #                     else :
+# #                         description+=rest[j]
+# #                 
+# #                 if description != "" :
+# #                     description = description.strip()
+# #                     view[i].set_description(description)
+#                 
+        elif l%4 == 1 :
+# #                
+            seq = line[:-1]
+#                 #print("seq", seq)
+            view[i].set_sequence(seq)
+            i+=1
+#                         
+        l+=1
+#     
+#     
+    input_file.close()
+
+    #print(view)
+    print(view.__repr__())
+
+    view.save_and_close()
+    d.close()
+
+    print("Done.")

python/obitools3/parsers/fasta.pxd

@@ -0,0 +1,8 @@
+#cython: language_level=3
+
+from .header cimport parseHeader
+from ..files.universalopener cimport uopen
+from ..files.linebuffer cimport LineBuffer
+
+
+    

python/obitools3/parsers/fasta.pyx

@@ -0,0 +1,46 @@
+#cython: language_level=3
+
+'''
+Created on 30 mars 2016
+
+@author: coissac
+'''
+
+def fastaIterator(lineiterator, int buffersize=100000000):
+    cdef LineBuffer lb
+    cdef str        ident
+    cdef str        definition
+    cdef dict       tags
+    cdef list       s
+    
+    if isinstance(lineiterator,(str,bytes)):
+        lineiterator=uopen(lineiterator)
+    
+    if isinstance(lineiterator, LineBuffer):
+        lb=lineiterator
+    else:
+        lb=LineBuffer(lineiterator,buffersize)
+        
+    i = iter(lb)
+    line = next(i)
+    
+    while True:
+        ident,tags,definition = parseHeader(line)
+        s = []
+        line = next(i)
+        while line[0]!='>':
+            s.append(line[0:-1])
+            line = next(i)
+        sequence  = "".join(s)
+        quality   = None
+
+        yield { "id"         : ident,
+                "definition" : definition,
+                "sequence"   : sequence,
+                "quality"    : quality,
+                "tags"       : tags,
+                "annotation" : {}
+              }
+        
+        
+        

python/obitools3/parsers/fastq.pxd

@@ -0,0 +1,8 @@
+#cython: language_level=3
+
+from .header cimport parseHeader
+from ..files.universalopener cimport uopen
+from ..files.linebuffer cimport LineBuffer
+
+
+    

python/obitools3/parsers/fastq.pyx

@@ -0,0 +1,41 @@
+#cython: language_level=3
+
+'''
+Created on 30 mars 2016
+
+@author: coissac
+'''
+
+
+
+def fastqIterator(lineiterator, int buffersize=100000000):
+    cdef LineBuffer lb
+    cdef str        ident
+    cdef str        definition
+    cdef dict       tags
+    
+    if isinstance(lineiterator,(str,bytes)):
+        lineiterator=uopen(lineiterator)
+    
+    if isinstance(lineiterator, LineBuffer):
+        lb=lineiterator
+    else:
+        lb=LineBuffer(lineiterator,buffersize)
+        
+    i = iter(lb)
+    for line in i:
+        ident,tags,definition = parseHeader(line)
+        sequence  = next(i)[0:-1]
+        next(i)
+        quality   = next(i)[0:-1]
+
+        yield { "id"         : ident,
+                "definition" : definition,
+                "sequence"   : sequence,
+                "quality"    : quality,
+                "tags"       : tags,
+                "annotation" : {}
+              }
+        
+        
+        

python/obitools3/parsers/header.pxd

@@ -0,0 +1,5 @@
+#cython: language_level=3
+
+cdef object __etag__(str x)
+
+cpdef tuple parseHeader(str header)

python/obitools3/parsers/header.pyx

@@ -0,0 +1,78 @@
+#cython: language_level=3
+
+'''
+Created on 25 mars 2016
+
+@author: coissac
+'''
+
+import re
+
+__ret__         = re.compile('''(([^ ]+)=('[^']*'|"[^"]*"|[^;]+); *)+?''')
+__re_int__      = re.compile("^[+-]?[0-9]+$")
+__re_float__    = re.compile("^[+-]?[0-9]+(\.[0-9]*)?([eE][+-]?[0-9]+)?$")
+__re_str__      = re.compile("""^"[^"]*"|'[^']*'$""")
+__re_dict__     = re.compile("""^\{\ *
+                                   (
+                                       ("[^"]*"|'[^']*')
+                                        \ *:\ *
+                                       ([^,}]+|
+                                        "[^"]*"|
+                                        '[^']*'
+                                       )
+                                   )?
+                                   (\ *,\ *
+                                       ("[^"]*"|'[^']*')
+                                        \ *:\ *
+                                       ([^,}]+|
+                                        "[^"]*"|
+                                        '[^']*'
+                                       )
+                                    )*\ *\}$""", re.VERBOSE)
+
+
+cdef object __etag__(str x):
+    if __re_int__.match(x):
+        v=int(x)
+    elif __re_float__.match(x):
+        v=float(x)
+    elif __re_str__.match(x):
+        v=x[1:-1]
+    elif x=='False':
+        v=False
+    elif x=='True':
+        v=True
+    elif __re_dict__.match(x):
+        v=eval(x)
+    else:
+        v=x
+    return v
+
+cpdef tuple parseHeader(str header):
+    cdef list m 
+    cdef dict tags 
+    cdef str definition
+    cdef str ident
+    cdef str second
+    
+    m=header[1:-1].split(maxsplit=1)
+
+    ident=m[0]
+    
+    if len(m)==1:
+        tags={}
+        definition='' 
+    else:
+        second=m[1]
+        m = __ret__.findall(second)
+        
+        if m:
+            tags = dict([(a[1],__etag__(a[2])) for a in m])
+            definition = second.split(m[-1][0],1)[1].strip()
+        else:
+            tags = {}
+            definition = second.strip() 
+        
+    return ident,tags,definition
+    
+    

python/obitools3/utils.pxd

@@ -1,5 +1,6 @@
 #cython: language_level=3
 
+
 cdef bytes str2bytes(str string)
 
 cdef str bytes2str(bytes string)

python/obitools3/utils.pyx

@@ -1,7 +1,28 @@
 #cython: language_level=3
 
+import sys
+import io
+
 cdef bytes str2bytes(str string):
+    """
+    Short cut to convert ascii encoded python string (str) to bytes 
+    which can be easily converted to C-strings. 
+    
+        @param string: the python string to be converted.
+        @type string: str
+        @return a transcoded string
+        @rtype: bytes 
+    """
     return string.encode('ascii')
 
 cdef str bytes2str(bytes string):
-    return string.decode('ascii')
+    """
+    Short cut to convert bytes (C-strings) to ascii encoded python string (str).
+    
+        @param string: the binary (C-string) string to be converted.
+        @type string: bytes
+        @return an ascii transcoded string
+        @rtype: str 
+    """
+    return string.decode('ascii')
+

requirements.txt

@@ -1,5 +1,5 @@
 --extra-index-url https://pypi.python.org/simple/
-Cython>=0.21
+Cython==0.23.5
 Sphinx>=1.2.0
 ipython>=3.0.0
 breathe>=4.0.0

src/bloom.c

@@ -0,0 +1,248 @@
+/*
+ *  Copyright (c) 2012-2015, Jyri J. Virkki
+ *  All rights reserved.
+ *
+ *  This file is under BSD license. See LICENSE file.
+ */
+
+/*
+ * Refer to bloom.h for documentation on the public interfaces.
+ */
+
+#include <assert.h>
+#include <fcntl.h>
+#include <math.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <unistd.h>
+
+#include "bloom.h"
+#include "murmurhash2.h"
+
+#define MAKESTRING(n) STRING(n)
+#define STRING(n) #n
+
+//#ifdef __linux__										// TODO commented because triggers error on luke21
+//unsigned detect_bucket_size(unsigned fallback_size);
+//#endif
+
+
+static int test_bit_set_bit(unsigned char * buf, unsigned int x, int set_bit)
+{
+  register uint32_t * word_buf = (uint32_t *)buf;
+  register unsigned int offset = x >> 5;
+  register uint32_t word = word_buf[offset];
+  register unsigned int mask = 1 << (x % 32);
+
+  if (word & mask) {
+    return 1;
+  } else {
+    if (set_bit) {
+      word_buf[offset] = word | mask;
+    }
+    return 0;
+  }
+}
+
+
+static int bloom_check_add(struct bloom * bloom,
+                           const void * buffer, int len, int add)
+{
+  if (bloom->ready == 0) {
+    (void)printf("bloom at %p not initialized!\n", (void *)bloom);
+    return -1;
+  }
+
+  int hits = 0;
+  register unsigned int a = murmurhash2(buffer, len, 0x9747b28c);
+  register unsigned int b = murmurhash2(buffer, len, a);
+  register unsigned int x;
+  register int i;	// TODO why was it unsigned?
+
+  unsigned bucket_index = (a % bloom->buckets);
+
+  unsigned char * bucket_ptr =
+    (bloom->bf + (bucket_index << bloom->bucket_bytes_exponent));
+
+  for (i = 0; i < bloom->hashes; i++) {
+    x = (a + i*b) & bloom->bucket_bits_fast_mod_operand;
+    if (test_bit_set_bit(bucket_ptr, x, add)) {
+      hits++;
+    }
+  }
+
+  if (hits == bloom->hashes) {
+    return 1;                // 1 == element already in (or collision)
+  }
+
+  return 0;
+}
+
+
+static void setup_buckets(struct bloom * bloom, unsigned int cache_size)
+{
+  // If caller passed a non-zero cache_size, use it as given, otherwise
+  // either compute it or use built-in default
+
+  if (cache_size == 0) {
+//#ifdef __linux__													// TODO commented because triggers error on luke21
+//    cache_size = detect_bucket_size(BLOOM_BUCKET_SIZE_FALLBACK);
+//#else
+    cache_size = BLOOM_BUCKET_SIZE_FALLBACK;
+//#endif
+  }
+
+  bloom->buckets = (bloom->bytes / cache_size);
+  bloom->bucket_bytes = cache_size;
+
+  // make sure bloom buffer bytes and bucket_bytes are even
+  int not_even_by = (bloom->bytes % bloom->bucket_bytes);
+
+  if (not_even_by) {
+    // adjust bytes
+    bloom->bytes += (bloom->bucket_bytes - not_even_by);
+    assert((bloom->bytes % bloom->bucket_bytes) == 0); // Should get even
+
+    // adjust bits
+    bloom->bits = bloom->bytes * 8;
+
+    // adjust bits per element
+    bloom->bpe = bloom->bits*1. / bloom->entries;
+
+    // adjust buckets
+    bloom->buckets++;
+  }
+
+  bloom->bucket_bytes_exponent = __builtin_ctz(cache_size);
+  bloom->bucket_bits_fast_mod_operand = (cache_size * 8 - 1);
+}
+
+
+int bloom_filter_size(int entries, double error)
+{
+	int bytes;
+	double num;
+	double denom;
+	double bpe;
+	int bits;
+	unsigned bucket_bytes;
+	int not_even_by;
+
+	num = log(error);
+	denom = 0.480453013918201; // ln(2)^2
+	bpe = -(num / denom);
+	bits = (int)(((double)entries) * bpe);
+
+	if (bits % 8) {
+		bytes = (bits / 8) + 1;
+	}
+	else {
+		bytes = bits / 8;
+	}
+
+	bucket_bytes = BLOOM_BUCKET_SIZE_FALLBACK;
+	not_even_by = bytes % bucket_bytes;
+	if (not_even_by) {
+	  // adjust bytes
+		bytes += (bucket_bytes - not_even_by);
+	}
+
+	return bytes;
+}
+
+
+int bloom_init_size(struct bloom * bloom, int entries, double error,
+                    unsigned int cache_size)
+{
+  bloom->ready = 0;
+
+  if (entries < 1 || error == 0) {
+    return 1;
+  }
+
+  bloom->entries = entries;
+  bloom->error = error;
+
+  double num = log(bloom->error);
+  double denom = 0.480453013918201; // ln(2)^2
+  bloom->bpe = -(num / denom);
+
+  double dentries = (double)entries;
+  bloom->bits = (int)(dentries * bloom->bpe);
+
+  if (bloom->bits % 8) {
+    bloom->bytes = (bloom->bits / 8) + 1;
+  } else {
+    bloom->bytes = bloom->bits / 8;
+  }
+
+  bloom->hashes = (int)ceil(0.693147180559945 * bloom->bpe);  // ln(2)
+
+  setup_buckets(bloom, cache_size);
+
+  // celine.mercier@metabarcoding.org :
+  // Replaced the calloc with a memset, as the memory for the bloom filter is mapped in our data structure
+  memset(bloom->bf, 0, bloom->bytes);
+  //bloom->bf = (unsigned char *)calloc(bloom->bytes, sizeof(unsigned char));
+  //if (bloom->bf == NULL) {
+  //  return 1;
+  //}
+
+  bloom->ready = 1;
+  return 0;
+}
+
+
+int bloom_init(struct bloom * bloom, int entries)  //, double error)
+{
+  return bloom_init_size(bloom, entries, BLOOM_FILTER_ERROR_RATE, 0);
+}
+
+
+int bloom_check(struct bloom * bloom, const void * buffer, int len)
+{
+  return bloom_check_add(bloom, buffer, len, 0);
+}
+
+
+int bloom_add(struct bloom * bloom, const void * buffer, int len)
+{
+  return bloom_check_add(bloom, buffer, len, 1);
+}
+
+
+void bloom_print(struct bloom * bloom)
+{
+  (void)printf("bloom at %p\n", (void *)bloom);
+  (void)printf(" ->entries = %d\n", bloom->entries);
+  (void)printf(" ->error = %f\n", bloom->error);
+  (void)printf(" ->bits = %d\n", bloom->bits);
+  (void)printf(" ->bits per elem = %f\n", bloom->bpe);
+  (void)printf(" ->bytes = %d\n", bloom->bytes);
+  (void)printf(" ->buckets = %u\n", bloom->buckets);
+  (void)printf(" ->bucket_bytes = %u\n", bloom->bucket_bytes);
+  (void)printf(" ->bucket_bytes_exponent = %u\n",
+               bloom->bucket_bytes_exponent);
+  (void)printf(" ->bucket_bits_fast_mod_operand = 0%o\n",
+               bloom->bucket_bits_fast_mod_operand);
+  (void)printf(" ->hash functions = %d\n", bloom->hashes);
+}
+
+
+void bloom_free(struct bloom * bloom)
+{
+  if (bloom->ready) {
+    free(bloom->bf);
+  }
+  bloom->ready = 0;
+}
+
+
+const char * bloom_version()
+{
+  return MAKESTRING(BLOOM_VERSION);
+}

src/bloom.h

@@ -0,0 +1,210 @@
+/*
+ *  Copyright (c) 2012-2015, Jyri J. Virkki
+ *  All rights reserved.
+ *
+ *  This file is under BSD license. See LICENSE file.
+ */
+
+#ifndef _BLOOM_H
+#define _BLOOM_H
+
+
+/** ***************************************************************************
+ * Added by celine.mercier@metabarcoding.org
+ *
+ * Bloom filter error rate wanted.
+ *
+ */
+#define BLOOM_FILTER_ERROR_RATE (0.001)
+
+
+/** ***************************************************************************
+ * On Linux, the code attempts to compute a bucket size based on CPU cache
+ * size info, if available. If that fails for any reason, this fallback size
+ * is used instead.
+ *
+ * On non-Linux systems, this is the bucket size always used unless the
+ * caller overrides it (see bloom_init_size()).
+ *
+ */
+#define BLOOM_BUCKET_SIZE_FALLBACK (32 * 1024)
+
+
+/** ***************************************************************************
+ * It was found that using multiplier x0.5 for CPU L1 cache size is
+ * more effective in terms of CPU usage and, surprisingly, collisions
+ * number.
+ *
+ * Feel free to tune this constant the way it will work for you.
+ *
+ */
+#define BLOOM_L1_CACHE_SIZE_DIV 1
+
+
+/** ***************************************************************************
+ * Structure to keep track of one bloom filter.  Caller needs to
+ * allocate this and pass it to the functions below. First call for
+ * every struct must be to bloom_init().
+ *
+ */
+struct bloom
+{
+  // These fields are part of the public interface of this structure.
+  // Client code may read these values if desired. Client code MUST NOT
+  // modify any of these.
+  int entries;
+  double error;
+  int bits;
+  int bytes;
+  int hashes;
+
+  // Fields below are private to the implementation. These may go away or
+  // change incompatibly at any moment. Client code MUST NOT access or rely
+  // on these.
+  unsigned buckets;
+  unsigned bucket_bytes;
+
+  // x86 CPU divide by/multiply by operation optimization helpers
+  unsigned bucket_bytes_exponent;
+  unsigned bucket_bits_fast_mod_operand;
+
+  double bpe;
+  int ready;
+
+  unsigned char bf[];
+};
+
+typedef struct bloom bloom_t;
+
+
+/** ***************************************************************************
+ * Added by celine.mercier@metabarcoding.org
+ *
+ * This function computes the size needed by the bloom filter
+ * depending on the number of entries and the error rate.
+ *
+ */
+int bloom_filter_size(int entries, double error);
+
+
+/** ***************************************************************************
+ * Initialize the bloom filter for use.
+ *
+ * The filter is initialized with a bit field and number of hash functions
+ * according to the computations from the wikipedia entry:
+ *     http://en.wikipedia.org/wiki/Bloom_filter
+ *
+ * Optimal number of bits is:
+ *     bits = (entries * ln(error)) / ln(2)^2
+ *
+ * Optimal number of hash functions is:
+ *     hashes = bpe * ln(2)
+ *
+ * Parameters:
+ * -----------
+ *     bloom   - Pointer to an allocated struct bloom (see above).
+ *     entries - The expected number of entries which will be inserted.
+ *     error   - Probability of collision (as long as entries are not
+ *               exceeded).
+ *
+ * Return:
+ * -------
+ *     0 - on success
+ *     1 - on failure
+ *
+ */
+int bloom_init(struct bloom * bloom, int entries); //, double error);
+
+
+/** ***************************************************************************
+ * Initialize the bloom filter for use.
+ *
+ * See comments above for general information.
+ *
+ * This is the same as bloom_init() but allows the caller to pass in a
+ * cache_size to override the internal value (which is either computed
+ * or the default of BLOOM_BUCKET_SIZE_FALLBACK). Mostly useful for
+ * experimenting.
+ *
+ * See misc/bucketsize for a script which can help identify a good value
+ * for cache_size.
+ *
+ */
+int bloom_init_size(struct bloom * bloom, int entries, double error,
+                    unsigned int cache_size);
+
+
+/** ***************************************************************************
+ * Check if the given element is in the bloom filter. Remember this may
+ * return false positive if a collision occured.
+ *
+ * Parameters:
+ * -----------
+ *     bloom  - Pointer to an allocated struct bloom (see above).
+ *     buffer - Pointer to buffer containing element to check.
+ *     len    - Size of 'buffer'.
+ *
+ * Return:
+ * -------
+ *     0 - element is not present
+ *     1 - element is present (or false positive due to collision)
+ *    -1 - bloom not initialized
+ *
+ */
+int bloom_check(struct bloom * bloom, const void * buffer, int len);
+
+
+/** ***************************************************************************
+ * Add the given element to the bloom filter.
+ * The return code indicates if the element (or a collision) was already in,
+ * so for the common check+add use case, no need to call check separately.
+ *
+ * Parameters:
+ * -----------
+ *     bloom  - Pointer to an allocated struct bloom (see above).
+ *     buffer - Pointer to buffer containing element to add.
+ *     len    - Size of 'buffer'.
+ *
+ * Return:
+ * -------
+ *     0 - element was not present and was added
+ *     1 - element (or a collision) had already been added previously
+ *    -1 - bloom not initialized
+ *
+ */
+int bloom_add(struct bloom * bloom, const void * buffer, int len);
+
+
+/** ***************************************************************************
+ * Print (to stdout) info about this bloom filter. Debugging aid.
+ *
+ */
+void bloom_print(struct bloom * bloom);
+
+
+/** ***************************************************************************
+ * Deallocate internal storage.
+ *
+ * Upon return, the bloom struct is no longer usable. You may call bloom_init
+ * again on the same struct to reinitialize it again.
+ *
+ * Parameters:
+ * -----------
+ *     bloom  - Pointer to an allocated struct bloom (see above).
+ *
+ * Return: none
+ *
+ */
+void bloom_free(struct bloom * bloom);
+
+
+/** ***************************************************************************
+ * Returns version string compiled into library.
+ *
+ * Return: version string
+ *
+ */
+const char * bloom_version();
+
+
+#endif

src/char_str_indexer.c

@@ -0,0 +1,80 @@
+/****************************************************************************
+ * Character string indexing functions                                      *
+ ****************************************************************************/
+
+/**
+ * @file char_str_indexer.c
+ * @author Celine Mercier
+ * @date April 12th 2016
+ * @brief Functions handling the indexing and retrieval of character strings.
+ */
+
+
+#include <stdlib.h>
+#include <stdio.h>
+#include <math.h>
+
+#include "obiblob.h"
+#include "obiblob_indexer.h"
+#include "obidebug.h"
+#include "obitypes.h"
+
+
+#define DEBUG_LEVEL 0	// TODO has to be defined somewhere else (cython compil flag?)
+
+
+Obi_blob_p obi_str_to_blob(const char* value)
+{
+	Obi_blob_p 		 value_b;
+	int32_t 		 length;
+
+	// Compute the number of bytes on which the value will be encoded
+	length = strlen(value) + 1;		// +1 to store \0 at the end (makes retrieving faster)
+
+	value_b = obi_blob((byte_t*)value, ELEMENT_SIZE_STR, length, length);
+	if (value_b == NULL)
+	{
+		obidebug(1, "\nError encoding a character string in a blob");
+		return NULL;
+	}
+
+	return value_b;
+}
+
+
+char* obi_blob_to_str(Obi_blob_p value_b)
+{
+	return value_b->value;
+}
+
+
+index_t obi_index_char_str(Obi_indexer_p indexer, const char* value)
+{
+	Obi_blob_p  value_b;
+	index_t 	idx;
+
+	// Encode value
+	value_b = obi_str_to_blob(value);
+	if (value_b == NULL)
+		return -1;
+
+	// Add in the indexer
+	idx = obi_indexer_add(indexer, value_b);
+
+	free(value_b);
+
+	return idx;
+}
+
+
+char* obi_retrieve_char_str(Obi_indexer_p indexer, index_t idx)
+{
+	Obi_blob_p  value_b;
+
+	// Get encoded value
+	value_b = obi_indexer_get(indexer, idx);
+
+	// Return decoded character string
+	return obi_blob_to_str(value_b);
+}
+

src/char_str_indexer.h

@@ -0,0 +1,87 @@
+/****************************************************************************
+ * DNA sequence indexer header file	                                        *
+ ****************************************************************************/
+
+/**
+ * @file dna_seq_indexer.h
+ * @author Celine Mercier
+ * @date April 12th 2016
+ * @brief Header file for the functions handling the indexing of DNA sequences.
+ */
+
+
+#ifndef CHAR_STR_INDEXER_H_
+#define CHAR_STR_INDEXER_H_
+
+
+#include <stdlib.h>
+#include <stdio.h>
+
+#include "obitypes.h"
+#include "obiblob.h"
+#include "obiblob_indexer.h"
+
+
+/**
+ * @brief Converts a character string to a blob.
+ *
+ * @warning The blob must be freed by the caller.
+ *
+ * @param value The character string to convert.
+ *
+ * @returns A pointer to the blob created.
+ * @retval NULL if an error occurred.
+ *
+ * @since October 2015
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+Obi_blob_p obi_str_to_blob(char* value);
+
+
+/**
+ * @brief Converts a blob to a character string.
+ *
+ * @warning The character string returned is mapped.
+ *
+ * @param value_b The blob to convert.
+ *
+ * @returns A pointer on the character string contained in the blob.
+ *
+ * @since October 2015
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+const char* obi_blob_to_str(Obi_blob_p value_b);
+
+
+/**
+ * @brief Stores a character string in an indexer and returns the index.
+ *
+ * @param indexer The indexer structure.
+ * @param value The character string to index.
+ *
+ * @returns The index referring to the stored character string in the indexer.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+index_t obi_index_char_str(Obi_indexer_p indexer, const char* value);
+
+
+/**
+ * @brief Retrieves a character string from an indexer.
+ *
+ * @warning The character string returned is mapped.
+ *
+ * @param indexer The indexer structure.
+ * @param idx The index referring to the character string to retrieve in the indexer.
+ *
+ * @returns A pointer on the character string.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+char* obi_retrieve_char_str(Obi_indexer_p indexer, index_t idx);
+
+
+#endif /* CHAR_STR_INDEXER_H_ */
+

src/crc64.c

@@ -0,0 +1,198 @@
+/* Redis uses the CRC64 variant with "Jones" coefficients and init value of 0.
+ *
+ * Specification of this CRC64 variant follows:
+ * Name: crc-64-jones
+ * Width: 64 bites
+ * Poly: 0xad93d23594c935a9
+ * Reflected In: True
+ * Xor_In: 0xffffffffffffffff
+ * Reflected_Out: True
+ * Xor_Out: 0x0
+ * Check("123456789"): 0xe9c6d914c4b8d9ca
+ *
+ * Copyright (c) 2012, Salvatore Sanfilippo <antirez at gmail dot com>
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE. */
+
+#include <stdint.h>
+
+static const uint64_t crc64_tab[256] = {
+    UINT64_C(0x0000000000000000), UINT64_C(0x7ad870c830358979),
+    UINT64_C(0xf5b0e190606b12f2), UINT64_C(0x8f689158505e9b8b),
+    UINT64_C(0xc038e5739841b68f), UINT64_C(0xbae095bba8743ff6),
+    UINT64_C(0x358804e3f82aa47d), UINT64_C(0x4f50742bc81f2d04),
+    UINT64_C(0xab28ecb46814fe75), UINT64_C(0xd1f09c7c5821770c),
+    UINT64_C(0x5e980d24087fec87), UINT64_C(0x24407dec384a65fe),
+    UINT64_C(0x6b1009c7f05548fa), UINT64_C(0x11c8790fc060c183),
+    UINT64_C(0x9ea0e857903e5a08), UINT64_C(0xe478989fa00bd371),
+    UINT64_C(0x7d08ff3b88be6f81), UINT64_C(0x07d08ff3b88be6f8),
+    UINT64_C(0x88b81eabe8d57d73), UINT64_C(0xf2606e63d8e0f40a),
+    UINT64_C(0xbd301a4810ffd90e), UINT64_C(0xc7e86a8020ca5077),
+    UINT64_C(0x4880fbd87094cbfc), UINT64_C(0x32588b1040a14285),
+    UINT64_C(0xd620138fe0aa91f4), UINT64_C(0xacf86347d09f188d),
+    UINT64_C(0x2390f21f80c18306), UINT64_C(0x594882d7b0f40a7f),
+    UINT64_C(0x1618f6fc78eb277b), UINT64_C(0x6cc0863448deae02),
+    UINT64_C(0xe3a8176c18803589), UINT64_C(0x997067a428b5bcf0),
+    UINT64_C(0xfa11fe77117cdf02), UINT64_C(0x80c98ebf2149567b),
+    UINT64_C(0x0fa11fe77117cdf0), UINT64_C(0x75796f2f41224489),
+    UINT64_C(0x3a291b04893d698d), UINT64_C(0x40f16bccb908e0f4),
+    UINT64_C(0xcf99fa94e9567b7f), UINT64_C(0xb5418a5cd963f206),
+    UINT64_C(0x513912c379682177), UINT64_C(0x2be1620b495da80e),
+    UINT64_C(0xa489f35319033385), UINT64_C(0xde51839b2936bafc),
+    UINT64_C(0x9101f7b0e12997f8), UINT64_C(0xebd98778d11c1e81),
+    UINT64_C(0x64b116208142850a), UINT64_C(0x1e6966e8b1770c73),
+    UINT64_C(0x8719014c99c2b083), UINT64_C(0xfdc17184a9f739fa),
+    UINT64_C(0x72a9e0dcf9a9a271), UINT64_C(0x08719014c99c2b08),
+    UINT64_C(0x4721e43f0183060c), UINT64_C(0x3df994f731b68f75),
+    UINT64_C(0xb29105af61e814fe), UINT64_C(0xc849756751dd9d87),
+    UINT64_C(0x2c31edf8f1d64ef6), UINT64_C(0x56e99d30c1e3c78f),
+    UINT64_C(0xd9810c6891bd5c04), UINT64_C(0xa3597ca0a188d57d),
+    UINT64_C(0xec09088b6997f879), UINT64_C(0x96d1784359a27100),
+    UINT64_C(0x19b9e91b09fcea8b), UINT64_C(0x636199d339c963f2),
+    UINT64_C(0xdf7adabd7a6e2d6f), UINT64_C(0xa5a2aa754a5ba416),
+    UINT64_C(0x2aca3b2d1a053f9d), UINT64_C(0x50124be52a30b6e4),
+    UINT64_C(0x1f423fcee22f9be0), UINT64_C(0x659a4f06d21a1299),
+    UINT64_C(0xeaf2de5e82448912), UINT64_C(0x902aae96b271006b),
+    UINT64_C(0x74523609127ad31a), UINT64_C(0x0e8a46c1224f5a63),
+    UINT64_C(0x81e2d7997211c1e8), UINT64_C(0xfb3aa75142244891),
+    UINT64_C(0xb46ad37a8a3b6595), UINT64_C(0xceb2a3b2ba0eecec),
+    UINT64_C(0x41da32eaea507767), UINT64_C(0x3b024222da65fe1e),
+    UINT64_C(0xa2722586f2d042ee), UINT64_C(0xd8aa554ec2e5cb97),
+    UINT64_C(0x57c2c41692bb501c), UINT64_C(0x2d1ab4dea28ed965),
+    UINT64_C(0x624ac0f56a91f461), UINT64_C(0x1892b03d5aa47d18),
+    UINT64_C(0x97fa21650afae693), UINT64_C(0xed2251ad3acf6fea),
+    UINT64_C(0x095ac9329ac4bc9b), UINT64_C(0x7382b9faaaf135e2),
+    UINT64_C(0xfcea28a2faafae69), UINT64_C(0x8632586aca9a2710),
+    UINT64_C(0xc9622c4102850a14), UINT64_C(0xb3ba5c8932b0836d),
+    UINT64_C(0x3cd2cdd162ee18e6), UINT64_C(0x460abd1952db919f),
+    UINT64_C(0x256b24ca6b12f26d), UINT64_C(0x5fb354025b277b14),
+    UINT64_C(0xd0dbc55a0b79e09f), UINT64_C(0xaa03b5923b4c69e6),
+    UINT64_C(0xe553c1b9f35344e2), UINT64_C(0x9f8bb171c366cd9b),
+    UINT64_C(0x10e3202993385610), UINT64_C(0x6a3b50e1a30ddf69),
+    UINT64_C(0x8e43c87e03060c18), UINT64_C(0xf49bb8b633338561),
+    UINT64_C(0x7bf329ee636d1eea), UINT64_C(0x012b592653589793),
+    UINT64_C(0x4e7b2d0d9b47ba97), UINT64_C(0x34a35dc5ab7233ee),
+    UINT64_C(0xbbcbcc9dfb2ca865), UINT64_C(0xc113bc55cb19211c),
+    UINT64_C(0x5863dbf1e3ac9dec), UINT64_C(0x22bbab39d3991495),
+    UINT64_C(0xadd33a6183c78f1e), UINT64_C(0xd70b4aa9b3f20667),
+    UINT64_C(0x985b3e827bed2b63), UINT64_C(0xe2834e4a4bd8a21a),
+    UINT64_C(0x6debdf121b863991), UINT64_C(0x1733afda2bb3b0e8),
+    UINT64_C(0xf34b37458bb86399), UINT64_C(0x8993478dbb8deae0),
+    UINT64_C(0x06fbd6d5ebd3716b), UINT64_C(0x7c23a61ddbe6f812),
+    UINT64_C(0x3373d23613f9d516), UINT64_C(0x49aba2fe23cc5c6f),
+    UINT64_C(0xc6c333a67392c7e4), UINT64_C(0xbc1b436e43a74e9d),
+    UINT64_C(0x95ac9329ac4bc9b5), UINT64_C(0xef74e3e19c7e40cc),
+    UINT64_C(0x601c72b9cc20db47), UINT64_C(0x1ac40271fc15523e),
+    UINT64_C(0x5594765a340a7f3a), UINT64_C(0x2f4c0692043ff643),
+    UINT64_C(0xa02497ca54616dc8), UINT64_C(0xdafce7026454e4b1),
+    UINT64_C(0x3e847f9dc45f37c0), UINT64_C(0x445c0f55f46abeb9),
+    UINT64_C(0xcb349e0da4342532), UINT64_C(0xb1eceec59401ac4b),
+    UINT64_C(0xfebc9aee5c1e814f), UINT64_C(0x8464ea266c2b0836),
+    UINT64_C(0x0b0c7b7e3c7593bd), UINT64_C(0x71d40bb60c401ac4),
+    UINT64_C(0xe8a46c1224f5a634), UINT64_C(0x927c1cda14c02f4d),
+    UINT64_C(0x1d148d82449eb4c6), UINT64_C(0x67ccfd4a74ab3dbf),
+    UINT64_C(0x289c8961bcb410bb), UINT64_C(0x5244f9a98c8199c2),
+    UINT64_C(0xdd2c68f1dcdf0249), UINT64_C(0xa7f41839ecea8b30),
+    UINT64_C(0x438c80a64ce15841), UINT64_C(0x3954f06e7cd4d138),
+    UINT64_C(0xb63c61362c8a4ab3), UINT64_C(0xcce411fe1cbfc3ca),
+    UINT64_C(0x83b465d5d4a0eece), UINT64_C(0xf96c151de49567b7),
+    UINT64_C(0x76048445b4cbfc3c), UINT64_C(0x0cdcf48d84fe7545),
+    UINT64_C(0x6fbd6d5ebd3716b7), UINT64_C(0x15651d968d029fce),
+    UINT64_C(0x9a0d8ccedd5c0445), UINT64_C(0xe0d5fc06ed698d3c),
+    UINT64_C(0xaf85882d2576a038), UINT64_C(0xd55df8e515432941),
+    UINT64_C(0x5a3569bd451db2ca), UINT64_C(0x20ed197575283bb3),
+    UINT64_C(0xc49581ead523e8c2), UINT64_C(0xbe4df122e51661bb),
+    UINT64_C(0x3125607ab548fa30), UINT64_C(0x4bfd10b2857d7349),
+    UINT64_C(0x04ad64994d625e4d), UINT64_C(0x7e7514517d57d734),
+    UINT64_C(0xf11d85092d094cbf), UINT64_C(0x8bc5f5c11d3cc5c6),
+    UINT64_C(0x12b5926535897936), UINT64_C(0x686de2ad05bcf04f),
+    UINT64_C(0xe70573f555e26bc4), UINT64_C(0x9ddd033d65d7e2bd),
+    UINT64_C(0xd28d7716adc8cfb9), UINT64_C(0xa85507de9dfd46c0),
+    UINT64_C(0x273d9686cda3dd4b), UINT64_C(0x5de5e64efd965432),
+    UINT64_C(0xb99d7ed15d9d8743), UINT64_C(0xc3450e196da80e3a),
+    UINT64_C(0x4c2d9f413df695b1), UINT64_C(0x36f5ef890dc31cc8),
+    UINT64_C(0x79a59ba2c5dc31cc), UINT64_C(0x037deb6af5e9b8b5),
+    UINT64_C(0x8c157a32a5b7233e), UINT64_C(0xf6cd0afa9582aa47),
+    UINT64_C(0x4ad64994d625e4da), UINT64_C(0x300e395ce6106da3),
+    UINT64_C(0xbf66a804b64ef628), UINT64_C(0xc5bed8cc867b7f51),
+    UINT64_C(0x8aeeace74e645255), UINT64_C(0xf036dc2f7e51db2c),
+    UINT64_C(0x7f5e4d772e0f40a7), UINT64_C(0x05863dbf1e3ac9de),
+    UINT64_C(0xe1fea520be311aaf), UINT64_C(0x9b26d5e88e0493d6),
+    UINT64_C(0x144e44b0de5a085d), UINT64_C(0x6e963478ee6f8124),
+    UINT64_C(0x21c640532670ac20), UINT64_C(0x5b1e309b16452559),
+    UINT64_C(0xd476a1c3461bbed2), UINT64_C(0xaeaed10b762e37ab),
+    UINT64_C(0x37deb6af5e9b8b5b), UINT64_C(0x4d06c6676eae0222),
+    UINT64_C(0xc26e573f3ef099a9), UINT64_C(0xb8b627f70ec510d0),
+    UINT64_C(0xf7e653dcc6da3dd4), UINT64_C(0x8d3e2314f6efb4ad),
+    UINT64_C(0x0256b24ca6b12f26), UINT64_C(0x788ec2849684a65f),
+    UINT64_C(0x9cf65a1b368f752e), UINT64_C(0xe62e2ad306bafc57),
+    UINT64_C(0x6946bb8b56e467dc), UINT64_C(0x139ecb4366d1eea5),
+    UINT64_C(0x5ccebf68aecec3a1), UINT64_C(0x2616cfa09efb4ad8),
+    UINT64_C(0xa97e5ef8cea5d153), UINT64_C(0xd3a62e30fe90582a),
+    UINT64_C(0xb0c7b7e3c7593bd8), UINT64_C(0xca1fc72bf76cb2a1),
+    UINT64_C(0x45775673a732292a), UINT64_C(0x3faf26bb9707a053),
+    UINT64_C(0x70ff52905f188d57), UINT64_C(0x0a2722586f2d042e),
+    UINT64_C(0x854fb3003f739fa5), UINT64_C(0xff97c3c80f4616dc),
+    UINT64_C(0x1bef5b57af4dc5ad), UINT64_C(0x61372b9f9f784cd4),
+    UINT64_C(0xee5fbac7cf26d75f), UINT64_C(0x9487ca0fff135e26),
+    UINT64_C(0xdbd7be24370c7322), UINT64_C(0xa10fceec0739fa5b),
+    UINT64_C(0x2e675fb4576761d0), UINT64_C(0x54bf2f7c6752e8a9),
+    UINT64_C(0xcdcf48d84fe75459), UINT64_C(0xb71738107fd2dd20),
+    UINT64_C(0x387fa9482f8c46ab), UINT64_C(0x42a7d9801fb9cfd2),
+    UINT64_C(0x0df7adabd7a6e2d6), UINT64_C(0x772fdd63e7936baf),
+    UINT64_C(0xf8474c3bb7cdf024), UINT64_C(0x829f3cf387f8795d),
+    UINT64_C(0x66e7a46c27f3aa2c), UINT64_C(0x1c3fd4a417c62355),
+    UINT64_C(0x935745fc4798b8de), UINT64_C(0xe98f353477ad31a7),
+    UINT64_C(0xa6df411fbfb21ca3), UINT64_C(0xdc0731d78f8795da),
+    UINT64_C(0x536fa08fdfd90e51), UINT64_C(0x29b7d047efec8728),
+};
+
+
+uint64_t crc64(const char* s, uint64_t l)
+{
+	uint64_t j;
+    uint64_t crc = 0;
+
+    for (j = 0; j < l; j++)
+    {
+        uint8_t byte = s[j];
+        crc = crc64_tab[(uint8_t)crc ^ byte] ^ (crc >> 8);
+    }
+    return crc;
+}
+
+
+/* Test main */
+//#ifdef TEST_MAIN
+//#include <stdio.h>
+//int main(void) {
+//    printf("e9c6d914c4b8d9ca == %016llx\n",
+//        (unsigned long long) crc64(0,(unsigned char*)"123456789",9));
+//    return 0;
+//}
+//#endif
+
+

src/crc64.h

@@ -0,0 +1,14 @@
+/**
+ * @file crc64.h
+ * @date March 24th 2016
+ * @brief Header file for CRC64 function.
+ */
+
+#include <stdint.h>
+
+
+/**
+ * @brief Generates and returns a 64-bit Cyclic Redundancy Check from a
+ *        character string s of length l.
+ */
+uint64_t crc64(const char* s, uint64_t l);

src/dna_seq_indexer.c

@@ -0,0 +1,102 @@
+/****************************************************************************
+ * DNA sequence indexing functions                                          *
+ ****************************************************************************/
+
+/**
+ * @file dna_seq_indexer.c
+ * @author Celine Mercier
+ * @date April 12th 2016
+ * @brief Functions handling the indexing and retrieval of DNA sequences.
+ */
+
+
+#include <stdlib.h>
+#include <stdio.h>
+#include <math.h>
+
+#include "obiblob.h"
+#include "obiblob_indexer.h"
+#include "obidebug.h"
+#include "obitypes.h"
+
+
+#define DEBUG_LEVEL 0	// TODO has to be defined somewhere else (cython compil flag?)
+
+
+Obi_blob_p obi_seq_to_blob(const char* seq)
+{
+	Obi_blob_p 		 value_b;
+	int32_t          length_encoded_seq;	// length of the encoded sequence in bytes
+	int32_t          seq_length;
+	byte_t*          encoded_seq;
+
+	seq_length = strlen(seq);
+
+	// Check if just ATGC and encode accordingly
+	if (only_ATGC(seq))
+	{
+		// Compute the length (in bytes) of the encoded sequence
+		length_encoded_seq = ceil((double) seq_length / (double) 4.0);
+		// Encode
+		encoded_seq = encode_seq_on_2_bits(seq, seq_length);
+		if (encoded_seq == NULL)
+			return NULL;
+		value_b = obi_blob(encoded_seq, ELEMENT_SIZE_SEQ_2, length_encoded_seq, seq_length);
+	}
+	else
+	{
+		// Compute the length (in bytes) of the encoded sequence
+		length_encoded_seq = ceil((double) seq_length / (double) 2.0);
+		// Encode
+		encoded_seq = encode_seq_on_4_bits(seq, seq_length);
+		if (encoded_seq == NULL)
+			return NULL;
+		value_b = obi_blob(encoded_seq, ELEMENT_SIZE_SEQ_4, length_encoded_seq, seq_length);
+	}
+
+	free(encoded_seq);
+
+	return value_b;
+}
+
+
+char* obi_blob_to_seq(Obi_blob_p value_b)
+{
+	// Decode
+	if (value_b->element_size == 2)
+		return decode_seq_on_2_bits(value_b->value, value_b->length_decoded_value);
+	else
+		return decode_seq_on_4_bits(value_b->value, value_b->length_decoded_value);
+}
+
+
+index_t obi_index_dna_seq(Obi_indexer_p indexer, const char* value)
+{
+	Obi_blob_p  value_b;
+	index_t 	idx;
+
+	// Encode value
+	value_b = obi_seq_to_blob(value);
+	if (value_b == NULL)
+		return -1;
+
+	// Add in the indexer
+	idx = obi_indexer_add(indexer, value_b);
+
+	free(value_b);
+
+	return idx;
+}
+
+
+char* obi_retrieve_dna_seq(Obi_indexer_p indexer, index_t idx)
+{
+	Obi_blob_p  value_b;
+
+	// Get encoded value
+	value_b = obi_indexer_get(indexer, idx);
+
+	// Return decoded sequence
+	return obi_blob_to_seq(value_b);
+}
+

src/dna_seq_indexer.h

@@ -0,0 +1,87 @@
+/****************************************************************************
+ * DNA sequence indexer header file	                                        *
+ ****************************************************************************/
+
+/**
+ * @file dna_seq_indexer.h
+ * @author Celine Mercier
+ * @date April 12th 2016
+ * @brief Header file for the functions handling the indexing of DNA sequences.
+ */
+
+
+#ifndef DNA_SEQ_INDEXER_H_
+#define DNA_SEQ_INDEXER_H_
+
+
+#include <stdlib.h>
+#include <stdio.h>
+
+#include "obidms.h"
+#include "obitypes.h"
+#include "obiblob.h"
+#include "obiblob_indexer.h"
+
+
+/**
+ * @brief Converts a DNA sequence to a blob.
+ *
+ * @warning The blob must be freed by the caller.
+ *
+ * @param value The DNA sequence to convert.
+ *
+ * @returns A pointer to the blob created.
+ * @retval NULL if an error occurred.
+ *
+ * @since November 2015
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+Obi_blob_p obi_seq_to_blob(const char* seq);
+
+
+/**
+ * @brief Converts a blob to a DNA sequence.
+ *
+ * @param value_b The blob to convert.
+ *
+ * @returns A pointer to the DNA sequence contained in the blob.
+ * @retval NULL if an error occurred.
+ *
+ * @since November 2015
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+char* obi_blob_to_seq(Obi_blob_p value_b);
+
+
+/**
+ * @brief Stores a DNA sequence in an indexer and returns the index.
+ *
+ * @param indexer The indexer structure.
+ * @param value The DNA sequence to index.
+ *
+ * @returns The index referring to the stored DNA sequence in the indexer.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+index_t obi_index_dna_seq(Obi_indexer_p indexer, const char* value);
+
+
+/**
+ * @brief Retrieves a DNA sequence from an indexer.
+ *
+ * @warning The DNA sequence returned must be freed by the caller.
+ *
+ * @param indexer The indexer structure.
+ * @param idx The index referring to the DNA sequence to retrieve in the indexer.
+ *
+ * @returns A pointer on the DNA sequence.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+char* obi_retrieve_dna_seq(Obi_indexer_p indexer, index_t idx);
+
+
+#endif /* DNA_SEQ_INDEXER_H_ */
+

src/encode.c

@@ -28,9 +28,9 @@
 
 
 
-bool only_ATGC(char* seq)
+bool only_ATGC(const char* seq)
 {
-	char* c = seq;
+	const char* c = seq;
 
 	while (*c)
 	{
@@ -54,7 +54,7 @@ bool only_ATGC(char* seq)
 }
 
 
-byte_t* encode_seq_on_2_bits(char* seq, int32_t length)
+byte_t* encode_seq_on_2_bits(const char* seq, int32_t length)
 {
 	byte_t*  seq_b;
 	uint8_t  modulo;
@@ -63,10 +63,13 @@ byte_t* encode_seq_on_2_bits(char* seq, int32_t length)
 
 	length_b = ceil((double) length / (double) 4.0);
 
-	seq_b = (byte_t*) malloc(length_b * sizeof(byte_t));
-
-	// Initialize all the bits to 0
-	memset(seq_b, 0, length_b);
+	seq_b = (byte_t*) calloc(length_b, sizeof(byte_t));
+	if (seq_b == NULL)
+	{
+		obi_set_errno(OBI_MALLOC_ERROR);
+		obidebug(1, "\nError allocating memory for an encoded DNA sequence");
+		return NULL;
+	}
 
 	for (i=0; i<length; i++)
 	{
@@ -93,6 +96,7 @@ byte_t* encode_seq_on_2_bits(char* seq, int32_t length)
 			seq_b[i/4] |= NUC_T_2b;
 			break;
 		default:
+			obi_set_errno(OBI_ENCODE_ERROR);
 			obidebug(1, "\nInvalid nucleotide base when encoding (not [atgcATGC])");
 			return NULL;
 		}
@@ -116,6 +120,12 @@ char* decode_seq_on_2_bits(byte_t* seq_b, int32_t length_seq)
 	uint8_t nuc;
 
 	seq = (char*) malloc((length_seq+1) * sizeof(char));
+	if (seq == NULL)
+	{
+		obi_set_errno(OBI_MALLOC_ERROR);
+		obidebug(1, "\nError allocating memory for a decoded DNA sequence");
+		return NULL;
+	}
 
 	for (i=0; i<length_seq; i++)
 	{
@@ -138,6 +148,7 @@ char* decode_seq_on_2_bits(byte_t* seq_b, int32_t length_seq)
 			seq[i] = 't';
 			break;
 		default:
+			obi_set_errno(OBI_DECODE_ERROR);
 			obidebug(1, "\nInvalid nucleotide base when decoding");
 			return NULL;
 		}
@@ -149,7 +160,7 @@ char* decode_seq_on_2_bits(byte_t* seq_b, int32_t length_seq)
 }
 
 
-byte_t* encode_seq_on_4_bits(char* seq, int32_t length)
+byte_t* encode_seq_on_4_bits(const char* seq, int32_t length)
 {
 	byte_t*  seq_b;
 	uint8_t  modulo;
@@ -158,10 +169,13 @@ byte_t* encode_seq_on_4_bits(char* seq, int32_t length)
 
 	length_b = ceil((double) length / (double) 2.0);
 
-	seq_b = (byte_t*) malloc(length_b * sizeof(byte_t));
-
-	// Initialize all the bits to 0
-	memset(seq_b, 0, length_b);
+	seq_b = (byte_t*) calloc(length_b, sizeof(byte_t));
+	if (seq_b == NULL)
+	{
+		obi_set_errno(OBI_MALLOC_ERROR);
+		obidebug(1, "\nError allocating memory for an encoded DNA sequence");
+		return NULL;
+	}
 
 	for (i=0; i<length; i++)
 	{
@@ -232,6 +246,7 @@ byte_t* encode_seq_on_4_bits(char* seq, int32_t length)
 			seq_b[i/2] |= NUC_N_4b;
 			break;
 		default:
+			obi_set_errno(OBI_ENCODE_ERROR);
 			obidebug(1, "\nInvalid nucleotide base when encoding (not IUPAC)");
 			return NULL;
 		}
@@ -255,6 +270,12 @@ char* decode_seq_on_4_bits(byte_t* seq_b, int32_t length_seq)
 	uint8_t nuc;
 
 	seq = (char*) malloc((length_seq+1) * sizeof(char));
+	if (seq == NULL)
+	{
+		obi_set_errno(OBI_MALLOC_ERROR);
+		obidebug(1, "\nError allocating memory for a decoded DNA sequence");
+		return NULL;
+	}
 
 	for (i=0; i<length_seq; i++)
 	{
@@ -310,6 +331,7 @@ char* decode_seq_on_4_bits(byte_t* seq_b, int32_t length_seq)
 			seq[i] = 'n';
 			break;
 		default:
+			obi_set_errno(OBI_DECODE_ERROR);
 			obidebug(1, "\nInvalid nucleotide base when decoding");
 			return NULL;
 		}

src/encode.h

@@ -10,6 +10,10 @@
  */
 
 
+#ifndef ENCODE_H_
+#define ENCODE_H_
+
+
 #include <stdlib.h>
 #include <stdio.h>
 #include <stdint.h>
@@ -18,8 +22,10 @@
 #include "obitypes.h"
 
 
-#define NUC_MASK_2B 0x3   /**< Binary: 11 to use when decoding 2 bits sequences */
-#define NUC_MASK_4B 0xF  /**< Binary: 1111 to use when decoding 4 bits sequences */
+#define NUC_MASK_2B 0x3   	/**< Binary: 11 to use when decoding 2 bits sequences
+ 	 	 	 	 	 	 	 */
+#define NUC_MASK_4B 0xF  	/**< Binary: 1111 to use when decoding 4 bits sequences
+ 	 	 	 	 	 	 	 */
 
 
 /**
@@ -69,7 +75,7 @@ enum
  * @since November 2015
  * @author Celine Mercier (celine.mercier@metabarcoding.org)
  */
-bool only_ATGC(char* seq);
+bool only_ATGC(const char* seq);
 
 
 /**
@@ -90,7 +96,7 @@ bool only_ATGC(char* seq);
  * @since November 2015
  * @author Celine Mercier (celine.mercier@metabarcoding.org)
  */
-byte_t* encode_seq_on_2_bits(char* seq, int32_t length);
+byte_t* encode_seq_on_2_bits(const char* seq, int32_t length);
 
 
 /**
@@ -101,7 +107,7 @@ byte_t* encode_seq_on_2_bits(char* seq, int32_t length);
  *    10 -> t
  *    11 -> g
  *
- * @param seq The sequence to decode.
+ * @param seq_b The sequence to decode.
  * @param length_seq The initial length of the sequence before it was encoded.
  *
  * @returns The decoded sequence ended with '\0'.
@@ -141,7 +147,7 @@ char* decode_seq_on_2_bits(byte_t* seq_b, int32_t length_seq);
  * @since November 2015
  * @author Celine Mercier (celine.mercier@metabarcoding.org)
  */
-byte_t* encode_seq_on_4_bits(char* seq, int32_t length);
+byte_t* encode_seq_on_4_bits(const char* seq, int32_t length);
 
 
 /**
@@ -163,7 +169,7 @@ byte_t* encode_seq_on_4_bits(char* seq, int32_t length);
  *      V or v : 1110
  *      N or n : 1111
  *
- * @param seq The sequence to decode.
+ * @param seq_b The sequence to decode.
  * @param length_seq The initial length of the sequence before it was encoded.
  *
  * @returns The decoded sequence ended with '\0'.
@@ -179,3 +185,6 @@ char* decode_seq_on_4_bits(byte_t* seq_b, int32_t length_seq);
 // little endian
 void print_bits(void* ptr, int32_t length);
 
+
+#endif /* ENCODE_H_ */
+

src/murmurhash2.c

@@ -0,0 +1,64 @@
+//-----------------------------------------------------------------------------
+// MurmurHash2, by Austin Appleby
+
+// Note - This code makes a few assumptions about how your machine behaves -
+
+// 1. We can read a 4-byte value from any address without crashing
+// 2. sizeof(int) == 4
+
+// And it has a few limitations -
+
+// 1. It will not work incrementally.
+// 2. It will not produce the same results on little-endian and big-endian
+//    machines.	->TODO
+
+unsigned int murmurhash2(const void * key, int len, const unsigned int seed)
+{
+	// 'm' and 'r' are mixing constants generated offline.
+	// They're not really 'magic', they just happen to work well.
+
+	const unsigned int m = 0x5bd1e995;
+	const int r = 24;
+
+	// Initialize the hash to a 'random' value
+
+	unsigned int h = seed ^ len;
+
+	// Mix 4 bytes at a time into the hash
+
+	const unsigned char * data = (const unsigned char *)key;
+
+	while(len >= 4)
+	{
+		unsigned int k = *(unsigned int *)data;
+
+		k *= m;
+		k ^= k >> r;
+		k *= m;
+
+		h *= m;
+		h ^= k;
+
+		data += 4;
+		len -= 4;
+	}
+
+	// Handle the last few bytes of the input array
+
+	switch(len)
+	{
+	case 3: h ^= data[2] << 16;
+	case 2: h ^= data[1] << 8;
+	case 1: h ^= data[0];
+	        h *= m;
+	};
+
+	// Do a few final mixes of the hash to ensure the last few
+	// bytes are well-incorporated.
+
+	h ^= h >> 13;
+	h *= m;
+	h ^= h >> 15;
+
+	return h;
+}

src/murmurhash2.h

@@ -0,0 +1,11 @@
+
+#ifndef _BLOOM_MURMURHASH2
+#define _BLOOM_MURMURHASH2
+
+
+/**
+ * @brief Generates and returns a hash code from a byte array.
+ */
+unsigned int murmurhash2(const void * key, int len, const unsigned int seed);
+
+#endif

src/obiavl.h

@@ -6,7 +6,7 @@
  * @file obiavl.h
  * @author Celine Mercier
  * @date December 3rd 2015
- * @brief Header file for handling AVL trees for storing and retrieving byte arrays (i.e. coding for character strings).
+ * @brief Header file for handling AVL trees for storing and retrieving blobs.
  */
 
 
@@ -23,35 +23,49 @@
 #include <stdbool.h>
 
 #include "obidms.h"
+#include "obiblob.h"
 #include "obitypes.h"
+#include "bloom.h"
+#include "utils.h"
+#include "encode.h"
 
 
+#define MAX_NB_OF_AVLS_IN_GROUP (100)						/**< The maximum number of AVL trees in a group.	// TODO discuss
+                              	  	 	 	 	 	 	 	 */
+#define MAX_NODE_COUNT_PER_AVL (10000000)					/**< The maximum number of nodes in an AVL tree.
+															 *   Only used to decide when to create a new AVL in a group, and to initialize the bloom filter // TODO discuss.
+                              	  	 	 	 	 	 	 	 */
+#define MAX_DATA_SIZE_PER_AVL (1073741824)		            /**< The maximum size of the data referred to by an AVL tree in a group.
+ 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 *   Only used to decide when to create a new AVL in a group.
+ 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 *   Should not be greater than int32_t max (2,147,483,647), as indexes will have to be stored on 32 bits.
+ 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 *   Here 1073741824 B = 1 GB
+ 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 */
+#define AVL_MAX_DEPTH (1024)								/**< The maximum depth of an AVL tree. Used to save paths through the tree.
+                              	  	 	 	 	 	 	 	 */
 #define AVL_MAX_NAME (1024) 								/**< The maximum length of an AVL tree name.
                               	  	 	 	 	 	 	 	 */
 #define AVL_GROWTH_FACTOR (2)								/**< The growth factor when an AVL tree is enlarged.
                                 	 	 	 	 	 	 	 */
-#define AVL_MAX_DEPTH (1000)								/**< The maximum depth of an AVL tree.
-                                	 	 	 	 	 	 	 */
 #define LEFT_CHILD(node)	(avl->tree)+(node->left_child)  /**< Pointer to the left child of a node in an AVL tree.
 															 */
 #define RIGHT_CHILD(node)	(avl->tree)+(node->right_child) /**< Pointer to the right child of a node in an AVL tree.
 															 */
-#define BYTE_ARRAY_HEADER_SIZE (9)  						/**< The size of the header of a byte array.
-                              	  	 	 	 	 	 	 	 */
 
 
 /**
  * @brief AVL tree node structure.
  */
 typedef struct AVL_node {
-	index_t left_child;		/**< Index of left less child node.
-	  	 	 	 	 	 	 */
-	index_t right_child; 	/**< Index of right greater child node.
-	 	 	 	 	 	 	 */
-	int8_t balance_factor; /**< Balance factor of the node.
-							 */
-	index_t value;			/**< Index of the value associated with the node in the data array.
-							 */
+	index_t  left_child;	  /**< Index of left less child node.
+	  	 	 	 	 	 	   */
+	index_t  right_child; 	  /**< Index of right greater child node.
+	 	 	 	 	 	 	   */
+	int8_t   balance_factor;  /**< Balance factor of the node.
+							   */
+	index_t  value;			  /**< Index of the value associated with the node in the data array.
+							   */
+	uint64_t crc64;			  /**< Cyclic Redundancy Check code on 64 bits associated with the value.
+	 	 	 	 	 	 	   */
 } AVL_node_t, *AVL_node_p;
 
 
@@ -80,8 +94,10 @@ typedef struct OBIDMS_avl_data_header {
 typedef struct OBIDMS_avl_data {
 	OBIDMS_avl_data_header_p	header; 		 	/**< A pointer to the header of the AVL tree data.
 	 	 	 	 	 	 	 	 	 	 	 	 	 */
-	byte_t*                   	data;   		 	/**< A pointer to the beginning of the data.
+	byte_t*                 	data;   		 	/**< A pointer to the beginning of the data.
 													 */
+	int							data_fd;	        /**< File descriptor of the file containing the data.
+	 	 	 	 	 	 	 	 	 	 	 	 	 */
 } OBIDMS_avl_data_t, *OBIDMS_avl_data_p;
 
 
@@ -103,6 +119,9 @@ typedef struct OBIDMS_avl_header {
 													 */
 	time_t		creation_date;			    	    /**< Date of creation of the file.
 												     */
+	bloom_t 	bloom_filter;			    	    /**< Bloom filter associated with the AVL tree, enabling to know if a value
+													 *   might already be stored in the data referred to by the tree.
+     	 	 	 	 	 	 	 	 	 	 	 	 */
 } OBIDMS_avl_header_t, *OBIDMS_avl_header_p;
 
 
@@ -110,40 +129,59 @@ typedef struct OBIDMS_avl_header {
  * @brief OBIDMS AVL tree structure.
  */
 typedef struct OBIDMS_avl {
-	OBIDMS_p                	dms;			 	/**< A pointer to the OBIDMS structure to which the AVL tree belongs.
-	 	 	 	 	 	 	 	 	 	 	 	 	 */
-	OBIDMS_avl_header_p			header; 	 		/**< A pointer to the header of the AVL tree.
-	 	 	 	 	 	 	 	 	 	 	 	 	 */
-	struct AVL_node*           	tree;   		 	/**< A pointer to the root of the AVL tree.
-													 */
+	OBIDMS_p                	dms;			 			/**< A pointer to the OBIDMS structure to which the AVL tree belongs.
+	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 */
+	OBIDMS_avl_header_p			header; 	 				/**< A pointer to the header of the AVL tree.
+	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 */
+	struct AVL_node*           	tree;   		 			/**< A pointer to the root of the AVL tree.
+	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 */
 	index_t                     path_idx[AVL_MAX_DEPTH];	/**< The path taken to a node from the root as an array of node indices.
 													 	     */
 	int8_t                      path_dir[AVL_MAX_DEPTH];	/**< The path taken to a node from the root as an array of directions
-															 * (0 for left, -1 for right).
+															 *   (0 for left, -1 for right).
 															 */
-	OBIDMS_avl_data_p			data;				/**< A pointer to the structure containing the data
-													 *   that the AVL tree references.
-	 	 	 	 	 	 	 	 	 	 	 	 	 */
-	DIR*						directory;			/**< A directory entry usable to
-										 	 	 	 *    refer and scan the AVL tree directory.
-										 	 	 	 */
-	int							dir_fd;				/**< The file descriptor of the directory entry
-													 *    usable to refer and scan the AVL tree directory.
-													 */
-	size_t						counter;			/**< Indicates by how many threads/programs (TODO) the AVL tree is used.
-													 */
+	OBIDMS_avl_data_p			data;						/**< A pointer to the structure containing the data
+													 	 	 *   that the AVL tree references.
+													 	 	 */
+	DIR*						directory;					/**< A directory entry usable to
+										 	 	 	 	 	 *    refer and scan the AVL tree directory.
+										 	 	 	 	 	 */
+	int							dir_fd;						/**< The file descriptor of the directory entry
+													 	 	 *    usable to refer and scan the AVL tree directory.
+													 	 	 */
+	int							avl_fd;						/**< The file descriptor of the file containing the AVL tree.
+													 	 	 */
 } OBIDMS_avl_t, *OBIDMS_avl_p;
 
 
 /**
- * @brief Checks if an AVL tree already exists or not.
+ * @brief OBIDMS AVL tree group structure.
+ */
+typedef struct OBIDMS_avl_group {
+	OBIDMS_avl_p 	sub_avls[MAX_NB_OF_AVLS_IN_GROUP];		/**< Array containing the pointers to the AVL trees of the group.
+	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 */
+	int 			current_avl_idx;						/**< Index in the sub_avls array of the AVL tree currently being filled.
+	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 */
+	char			name[AVL_MAX_NAME+1];				 	/**< Base name of the AVL group. The AVL trees in it have names of the form basename_idx.
+	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 */
+	OBIDMS_p        dms;									/**< Pointer to the OBIDMS structure to which the AVL group belongs.
+	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 */
+	bool			writable;								/**< Indicates whether the AVL group is read-only or not.
+															 */
+	size_t			counter;								/**< Indicates by how many threads/programs (TODO) the AVL group is used.
+													 	 	 */
+} OBIDMS_avl_group_t, *OBIDMS_avl_group_p;
+
+
+/**
+ * @brief Checks if an AVL tree or AVL tree group already exists or not.
  *
- * @param dms The OBIDMS to which the AVL tree belongs.
- * @param avl_name The name of the AVL tree.
+ * @param dms The OBIDMS to which the AVL tree or AVL tree group belongs.
+ * @param avl_name The name of the AVL treeor the base name of the AVL tree group.
  *
- * @returns A value indicating whether the AVL tree exists or not.
- * @retval 1 if the AVL tree exists.
- * @retval 0 if the AVL tree does not exist.
+ * @returns A value indicating whether the AVL tree or AVL tree group exists or not.
+ * @retval 1 if the AVL tree or AVL tree group exists.
+ * @retval 0 if the AVL tree or AVL tree group does not exist.
  * @retval -1 if an error occurred.
  *
  * @since December 2015
@@ -152,36 +190,20 @@ typedef struct OBIDMS_avl {
 int obi_avl_exists(OBIDMS_p dms, const char* avl_name);
 
 
-/**
- * @brief Opens an AVL tree and creates it if it does not already exist.
- *
- * Note: An AVL tree is made of two files (referred to by two structures).
- * 		 One file contains the indices referring to the data, and the other
- * 		 file contains the data itself. The AVL tree as a whole is referred
- * 		 to via the OBIDMS_avl structure.
- *
- * @param dms The OBIDMS to which the AVL tree belongs.
- * @param avl_name The name of the AVL tree.
- *
- * @returns A pointer to the AVL tree structure.
- * @retval NULL if an error occurred.
- *
- * @since December 2015
- * @author Celine Mercier (celine.mercier@metabarcoding.org)
- */
-OBIDMS_avl_p obi_avl(OBIDMS_p dms, const char* avl_name);
-
-
 /**
  * @brief Creates an AVL tree. Fails if it already exists.
  *
  * Note: An AVL tree is made of two files (referred to by two structures).
- * 		 One file contains the indices referring to the data, and the other
+ * 		 One file contains the tree referring to the data, and the other
  * 		 file contains the data itself. The AVL tree as a whole is referred
- * 		 to via the OBIDMS_avl structure.
+ * 		 to via the OBIDMS_avl structure. An AVL tree is stored in a directory
+ * 		 with the same name, or with the base name of the AVL group if it is
+ * 		 part of an AVL group.
  *
  * @param dms The OBIDMS to which the AVL tree belongs.
  * @param avl_name The name of the AVL tree.
+ * @param avl_idx The index of the AVL tree if it is part of an AVL group,
+ *        or -1 if it is not part of an AVL group.
  *
  * @returns A pointer to the newly created AVL tree structure.
  * @retval NULL if an error occurred.
@@ -189,19 +211,21 @@ OBIDMS_avl_p obi_avl(OBIDMS_p dms, const char* avl_name);
  * @since December 2015
  * @author Celine Mercier (celine.mercier@metabarcoding.org)
  */
-OBIDMS_avl_p obi_create_avl(OBIDMS_p dms, const char* avl_name);
+OBIDMS_avl_p obi_create_avl(OBIDMS_p dms, const char* avl_name, int avl_idx);
 
 
 /**
- * @brief Opens an AVL tree. Fails if it does not already exist.
+ * @brief Opens an AVL tree in read-only mode. Fails if it does not already exist.
  *
  * Note: An AVL tree is made of two files (referred to by two structures).
- * 		 One file contains the indices referring to the data, and the other
+ * 		 One file contains the tree referring to the data, and the other
  * 		 file contains the data itself. The AVL tree as a whole is referred
  * 		 to via the OBIDMS_avl structure.
  *
  * @param dms The OBIDMS to which the AVL tree belongs.
  * @param avl_name The name of the AVL tree.
+ * @param avl_idx The index of the AVL tree if it is part of an AVL group,
+ *        or -1 if it is not part of an AVL group.
  *
  * @returns A pointer to the AVL tree structure.
  * @retval NULL if an error occurred.
@@ -209,17 +233,66 @@ OBIDMS_avl_p obi_create_avl(OBIDMS_p dms, const char* avl_name);
  * @since December 2015
  * @author Celine Mercier (celine.mercier@metabarcoding.org)
  */
-OBIDMS_avl_p obi_open_avl(OBIDMS_p dms, const char* avl_name);
+OBIDMS_avl_p obi_open_avl(OBIDMS_p dms, const char* avl_name, int avl_idx);
+
+
+/**
+ * @brief Opens an AVL tree group and creates it if it does not already exist.
+ *
+ * Note: An AVL tree group is composed of multiple AVL trees that all have the
+ *       same base name, and an index differentiating them.
+ *
+ * @param dms The OBIDMS to which the AVL tree belongs.
+ * @param avl_name The base name of the AVL tree group.
+ *
+ * @returns A pointer to the AVL tree group structure.
+ * @retval NULL if an error occurred.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+OBIDMS_avl_group_p obi_avl_group(OBIDMS_p dms, const char* avl_name);
+
+
+/**
+ * @brief Creates an AVL tree group.
+ *
+ * Note: An AVL tree group is composed of multiple AVL trees that all have the
+ *       same base name, and an index differentiating them.
+ *
+ * @param dms The OBIDMS to which the AVL tree belongs.
+ * @param avl_name The base name of the AVL tree group.
+ *
+ * @returns A pointer to the AVL tree group structure.
+ * @retval NULL if an error occurred.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+OBIDMS_avl_group_p obi_create_avl_group(OBIDMS_p dms, const char* avl_name);
+
+
+/**
+ * @brief Opens an AVL tree group in read-only mode.
+ *
+ * Note: An AVL tree group is composed of multiple AVL trees that all have the
+ *       same base name, and an index differentiating them.
+ *
+ * @param dms The OBIDMS to which the AVL tree belongs.
+ * @param avl_name The base name of the AVL tree group.
+ *
+ * @returns A pointer to the AVL tree group structure.
+ * @retval NULL if an error occurred.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+OBIDMS_avl_group_p obi_open_avl_group(OBIDMS_p dms, const char* avl_name);
 
 
 /**
  * @brief Closes an AVL tree.
  *
- * Note: An AVL tree is made of two files (referred to by two structures).
- * 		 One file contains the indices referring to the data, and the other
- * 		 file contains the data itself. The AVL tree as a whole is referred
- * 		 to via the OBIDMS_avl structure.
- *
  * @param avl A pointer to the AVL tree structure to close and free.
  *
  * @retval 0 if the operation was successfully completed.
@@ -232,29 +305,61 @@ int obi_close_avl(OBIDMS_avl_p avl);
 
 
 /**
- * @brief Adds a value (byte array) in an AVL tree, checking if it is already in it.
+ * @brief Closes an AVL tree group.
  *
- * @warning The byte array to add must already be encoded and contain its header.
+ * @param avl_group A pointer to the AVL tree group structure to close and free.
+ *
+ * @retval 0 if the operation was successfully completed.
+ * @retval -1 if an error occurred.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+int obi_close_avl_group(OBIDMS_avl_group_p avl_group);
+
+
+/**
+ * @brief Recovers a value (blob) in an AVL tree.
+ *
+ * @warning The blob recovered must be decoded to get the original value.
+ * @warning The blob recovered is mapped in memory.
  *
  * @param avl A pointer to the AVL tree.
- * @param value The byte array to add in the AVL tree.
+ * @param index The index of the value in the data array.
  *
- * @returns The index of the value, whether it was added or already in the AVL tree.
+ * @returns A pointer to the blob recovered.
+ *
+ * @since December 2015
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+Obi_blob_p obi_avl_get(OBIDMS_avl_p avl, index_t index);
+
+
+/**
+ * @brief Adds a value (blob) in an AVL tree.
+ *
+ * @warning The value given must be already be encoded into a blob structure (Obi_blob_t).
+ * @warning If the value is already in the AVL tree, an error will be triggered.		// TODO to discuss
+ *
+ * @param avl A pointer to the AVL tree.
+ * @param value The blob to add in the AVL tree.
+ *
+ * @returns The index of the value newly added in the AVL tree.
  * @retval -1 if an error occurred.
  *
  * @since December 2015
  * @author Celine Mercier (celine.mercier@metabarcoding.org)
  */
-index_t obi_avl_add(OBIDMS_avl_p avl, byte_t* value);
+index_t obi_avl_add(OBIDMS_avl_p avl, Obi_blob_p value);
 
 
 /**
- * @brief Finds a value (byte array) in an AVL tree, checking first if it is already in it.
+ * @brief Finds a value (blob) in an AVL tree.
  *
- * @warning The byte array to add must already be encoded and contain its header.
+ * @warning The value given must be already be encoded into a blob structure (Obi_blob_t).
  *
  * @param avl A pointer to the AVL tree.
- * @param value The byte array to add in the AVL tree.
+ * @param value The blob to add in the AVL tree.
  *
  * @returns The data index of the value.
  * @retval -1 if the value is not in the tree.
@@ -262,82 +367,60 @@ index_t obi_avl_add(OBIDMS_avl_p avl, byte_t* value);
  * @since December 2015
  * @author Celine Mercier (celine.mercier@metabarcoding.org)
  */
-index_t obi_avl_find(OBIDMS_avl_p avl, byte_t* value);
+index_t obi_avl_find(OBIDMS_avl_p avl, Obi_blob_p value);
 
 
 /**
- * @brief Recovers a value (byte array) in an AVL tree.
+ * @brief Recovers a value (blob) in an AVL tree.
  *
- * @warning The byte array recovered is encoded and contains its header.
+ * @warning The blob recovered must be decoded to get the original value.
+ * @warning The blob recovered is mapped in memory.
  *
- * @param avl A pointer to the AVL tree.
- * @param index The index of the value in the data array.
+ * @param avl_group A pointer to the AVL tree.
+ * @param index The index of the value in the form of a 64-bit integer
+ *              with the 32 left-most bits coding for the index of the tree of
+ *              the group in which the value is stored, and the 32 right-most bits
+ *              coding for the index at which the value is stored in that AVL tree.
  *
- * @returns A pointer to the byte array recovered.
+ * @returns A pointer to the blob recovered.
  *
- * @since December 2015
+ * @since April 2016
  * @author Celine Mercier (celine.mercier@metabarcoding.org)
  */
-byte_t* obi_avl_get(OBIDMS_avl_p avl, index_t index);
+Obi_blob_p obi_avl_group_get(OBIDMS_avl_group_p avl_group, index_t idx);
 
 
 /**
- * @brief Converts a character string to a byte array with a header.
+ * @brief Adds a value (blob) in an AVL tree group, checking if it is already in it.
  *
- * @warning The byte array must be freed by the caller.
+ * @warning The value given must be already be encoded into a blob structure (Obi_blob_t).
  *
- * @param value The character string to convert.
+ * @param avl_group A pointer to the AVL tree group.
+ * @param value The blob to add in the AVL tree group.
  *
- * @returns A pointer to the byte array created.
- * @retval NULL if an error occurred.
+ * @returns The index of the value newly added in the AVL tree group, in the form of a
+ *          64-bit integer with the 32 left-most bits coding for the index of the tree
+ *          of the group in which the value is stored, and the 32 right-most bits
+ *          coding for the index at which the value is stored in that AVL tree.
+ * @retval -1 if an error occurred.
  *
- * @since October 2015
+ * @since April 2016
  * @author Celine Mercier (celine.mercier@metabarcoding.org)
  */
-byte_t* obi_str_to_obibytes(char* value);
+index_t obi_avl_group_add(OBIDMS_avl_group_p avl_group, Obi_blob_p value);
 
 
 /**
- * @brief Converts a byte array to a character string.
+ * @brief Recovers the name of an AVL group.
  *
- * @param value_b The byte array to convert.
+ * @param avl_group A pointer on the AVL group structure.
  *
- * @returns A pointer to the character string contained in the byte array.
+ * @returns A pointer on the name of the AVL group.
  *
- * @since October 2015
+ * @since April 2016
  * @author Celine Mercier (celine.mercier@metabarcoding.org)
  */
-const char* obi_obibytes_to_str(byte_t* value_b);
-
-
-/**
- * @brief Converts a DNA sequence to a byte array with a header.
- *
- * @warning The byte array must be freed by the caller.
- *
- * @param value The DNA sequence to convert.
- *
- * @returns A pointer to the byte array created.
- * @retval NULL if an error occurred.
- *
- * @since November 2015
- * @author Celine Mercier (celine.mercier@metabarcoding.org)
- */
-byte_t* obi_seq_to_obibytes(char* seq);
-
-
-/**
- * @brief Converts a byte array to a DNA sequence.
- *
- * @param value_b The byte array to convert.
- *
- * @returns A pointer to the DNA sequence contained in the byte array.
- * @retval NULL if an error occurred.
- *
- * @since November 2015
- * @author Celine Mercier (celine.mercier@metabarcoding.org)
- */
-const char* obi_obibytes_to_seq(byte_t* value_b);
+const char* obi_avl_group_get_name(OBIDMS_avl_group_p avl_group);
 
 
 #endif /* OBIAVL_H_ */

src/obiblob.c

@@ -0,0 +1,91 @@
+/****************************************************************************
+ * Obiblob functions                                                       *
+ ****************************************************************************/
+
+/**
+ * @file obiblob.c
+ * @author Celine Mercier
+ * @date April 11th 2016
+ * @brief Functions handling Obiblob structures.
+ */
+
+
+#include <stdlib.h>
+#include <stdio.h>
+#include <stdint.h>
+#include <math.h>
+
+#include "obiblob.h"
+#include "obierrno.h"
+#include "obitypes.h"	// For byte_t type
+#include "obidebug.h"
+
+
+#define DEBUG_LEVEL 0	// TODO has to be defined somewhere else (cython compil flag?)
+
+
+// TODO: endianness problem?
+
+
+Obi_blob_p obi_blob(byte_t* encoded_value, uint8_t element_size, int32_t length_encoded_value, int32_t length_decoded_value)
+{
+	Obi_blob_p blob;
+
+	// Allocate the memory for the blob structure
+	blob = (Obi_blob_p) malloc(sizeof(Obi_blob_t) + length_encoded_value);
+	if (blob == NULL)
+	{
+		obi_set_errno(OBI_MALLOC_ERROR);
+		obidebug(1, "\nError allocating memory for a blob");
+		return NULL;
+	}
+
+	// Store the number of bits on which each element is encoded
+	blob->element_size = element_size;
+
+	// Store the length (in bytes) of the encoded value
+	blob->length_encoded_value = length_encoded_value;
+
+	// Store the initial length (in bytes) of the decoded value
+	blob->length_decoded_value = length_decoded_value;
+
+	// Store the encoded value
+	memcpy(blob->value, encoded_value, length_encoded_value);
+
+	return blob;
+}
+
+
+int obi_blob_compare(Obi_blob_p value_1, Obi_blob_p value_2)
+{
+	int 	comp;
+	int32_t b;
+
+	if (value_1->element_size != value_2->element_size)
+		return (value_1->element_size - value_2->element_size);
+
+	if (value_1->length_encoded_value != value_2->length_encoded_value)
+		return (value_1->length_encoded_value - value_2->length_encoded_value);
+
+	if (value_1->element_size != ELEMENT_SIZE_STR) // because if so, length_decoded_value == length_encoded_value
+	{
+		if (value_1->length_decoded_value != value_2->length_decoded_value)
+			return (value_1->length_decoded_value - value_2->length_decoded_value);
+	}
+
+	b = 0;
+	comp = 0;
+	while (!comp && (b < value_1->length_encoded_value))
+	{
+		comp = *((value_1->value)+b) - *((value_2->value)+b);
+		b++;
+	}
+	return comp;
+}
+
+
+int obi_blob_sizeof(Obi_blob_p value)
+{
+	return (sizeof(Obi_blob_t) + (value->length_encoded_value));
+}
+

src/obiblob.h

@@ -0,0 +1,98 @@
+/****************************************************************************
+ * Obiblob header file	                                                    *
+ ****************************************************************************/
+
+/**
+ * @file obiblob.h
+ * @author Celine Mercier
+ * @date November 18th 2015
+ * @brief Header file for handling Obi_blob structures.
+ */
+
+
+#ifndef OBIBLOB_H_
+#define OBIBLOB_H_
+
+
+#include <stdlib.h>
+#include <stdio.h>
+#include <stdint.h>
+
+#include "obitypes.h"
+
+
+#define ELEMENT_SIZE_STR (8)		/**< The size of an element from a value of type character string.
+                              	  	 */
+#define ELEMENT_SIZE_SEQ_2 (2)		/**< The size of an element from a value of type DNA sequence encoded on 2 bits.
+                              	  	 */
+#define ELEMENT_SIZE_SEQ_4 (4)		/**< The size of an element from a value of type DNA sequence encoded on 4 bits.
+                              	  	 */
+
+
+/**
+ * @brief Blob structure, for handling encoded values.
+ */
+typedef struct Obi_blob {
+	uint8_t  element_size;		 	/**< Size in bits of one element from the encoded value.
+	 	 	 	 	 	 	 	 	 */
+	int32_t  length_encoded_value;	/**< Length in bytes of the encoded value.
+	 	 	 	 	 	 	 	 	 */
+	int32_t  length_decoded_value;	/**< Length in bytes of the decoded value.
+	 	 	 	 	 	 	 	 	 */
+	byte_t   value[];				/**< Encoded value.
+	 	 	 	 	 	 	 	 	 */
+} Obi_blob_t, *Obi_blob_p;
+
+
+
+/**
+ * @brief Function building a blob structure.
+ *
+ * @param encoded_value A pointer to the encoded value that will be stored in the blob.
+ * @param element_size The size in bits of one element from the encoded value.
+ * @param length_encoded_value The length in bytes of the encoded value.
+ * @param length_decoded_value The length in bytes of the decoded value.
+ *
+ * @returns A pointer to the created blob structure.
+ * @retval NULL if an error occurred.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+Obi_blob_p obi_blob(byte_t* encoded_value, uint8_t element_size, int32_t length_encoded_value, int32_t length_decoded_value);
+
+
+/**
+ * @brief Function comparing two blobs.
+ *
+ * The encoding is compared first, then the length of the
+ * values, then the values themselves.
+ *
+ * @param value_1 A pointer to the first blob structure.
+ * @param value_2 A pointer to the second blob structure.
+ *
+ * @returns A value < 0 if value_1 < value_2,
+ * 			a value > 0 if value_1 > value_2,
+ * 			and 0 if value_1 == value_2.
+ *
+ * @since October 2015
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+int obi_blob_compare(Obi_blob_p value_1, Obi_blob_p value_2);
+
+
+/**
+ * @brief Function calculating the size in bytes of a blob.
+ *
+ * @param value A pointer to the blob structure.
+ *
+ * @returns The size of the blob in bytes.
+ *
+ * @since October 2015
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+int obi_blob_sizeof(Obi_blob_p value);
+
+
+#endif /* OBIBLOB_H_ */
+

src/obiblob_indexer.c

@@ -0,0 +1,59 @@
+/****************************************************************************
+ * Obiblob functions                                                       *
+ ****************************************************************************/
+
+/**
+ * @file obiblob_indexer.c
+ * @author Celine Mercier
+ * @date April 12th 2016
+ * @brief Functions handling the indexing and retrieval of blob structures.
+ */
+
+
+#include <stdlib.h>
+#include <stdio.h>
+
+#include "obiblob_indexer.h"
+#include "obidms.h"
+#include "obierrno.h"
+#include "obidebug.h"
+
+
+#define DEBUG_LEVEL 0	// TODO has to be defined somewhere else (cython compil flag?)
+
+
+inline int obi_indexer_exists(OBIDMS_p dms, const char* name);
+
+inline Obi_indexer_p obi_indexer(OBIDMS_p dms, const char* name);
+
+inline Obi_indexer_p obi_create_indexer(OBIDMS_p dms, const char* name);
+
+inline Obi_indexer_p obi_open_indexer(OBIDMS_p dms, const char* name);
+
+inline int obi_close_indexer(Obi_indexer_p indexer);
+
+inline index_t obi_indexer_add(Obi_indexer_p indexer, Obi_blob_p value);
+
+inline Obi_blob_p obi_indexer_get(Obi_indexer_p indexer, index_t idx);
+
+inline const char* obi_indexer_get_name(Obi_indexer_p indexer);
+
+
+char* obi_build_indexer_name(const char* column_name, obiversion_t column_version)
+{
+	char* indexer_name;
+
+	indexer_name = (char*) malloc(INDEXER_MAX_NAME * sizeof(char));
+	if (indexer_name == NULL)
+	{
+		obi_set_errno(OBI_MALLOC_ERROR);
+		obidebug(1, "\nError allocating the memory for an indexer name");
+		return NULL;
+	}
+
+	strcpy(indexer_name, column_name);
+	sprintf(indexer_name+strlen(column_name), "_%d_indexer", column_version);
+
+	return indexer_name;
+}
+

src/obiblob_indexer.h

@@ -0,0 +1,196 @@
+/****************************************************************************
+ * Blob indexer header file	                                                    *
+ ****************************************************************************/
+
+/**
+ * @file obiblob_indexer.h
+ * @author Celine Mercier
+ * @date April 12th 2016
+ * @brief Header file for the functions handling the indexing of values.
+ */
+
+
+#ifndef OBIBLOB_INDEXER_H_
+#define OBIBLOB_INDEXER_H_
+
+
+#include <stdlib.h>
+#include <stdio.h>
+
+#include "obidms.h"
+#include "obiavl.h"
+#include "obitypes.h"
+#include "obiblob.h"
+
+
+#define INDEXER_MAX_NAME AVL_MAX_NAME 			/**< Macro to refer to the maximum size of the name of an indexer structure.
+ 	 	 	 	 	 	 	 	 	 	 	 	 */
+
+
+typedef struct OBIDMS_avl_group Obi_indexer;	/**< Typedef to refer to the used indexer structure.
+ 	 	 	 	 	 	 	 	 	 	 	 	 */
+typedef OBIDMS_avl_group_p Obi_indexer_p; 		/**< Typedef to refer to the pointer of the used indexer structure.
+ 	 	 	 	 	 	 	 	 	 	 	 	 */
+
+
+/**
+ * @brief Checks if an indexer already exists or not.
+ *
+ * @param dms The OBIDMS to which the indexer belongs.
+ * @param name The name of the indexer.
+ *
+ * @returns A value indicating whether the indexer exists or not.
+ * @retval 1 if the indexer exists.
+ * @retval 0 if the indexer does not exist.
+ * @retval -1 if an error occurred.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+inline int obi_indexer_exists(OBIDMS_p dms, const char* name)
+{
+	return obi_avl_exists(dms, name);
+}
+
+
+/**
+ * @brief Opens an indexer and creates it if it does not already exist.
+ *
+ * @param dms The OBIDMS to which the indexer belongs.
+ * @param name The name of the indexer.
+ *
+ * @returns A pointer to the indexer structure.
+ * @retval NULL if an error occurred.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+inline Obi_indexer_p obi_indexer(OBIDMS_p dms, const char* name)
+{
+	return obi_avl_group(dms, name);
+}
+
+
+/**
+ * @brief Creates an indexer.
+ *
+ * @param dms The OBIDMS to which the indexer belongs.
+ * @param name The name of the indexer.
+ *
+ * @returns A pointer to the AVL tree group structure.
+ * @retval NULL if an error occurred.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+inline Obi_indexer_p obi_create_indexer(OBIDMS_p dms, const char* name)
+{
+	return obi_create_avl_group(dms, name);
+}
+
+
+/**
+ * @brief Opens an indexer.
+ *
+ * @param dms The OBIDMS to which the indexer belongs.
+ * @param name The name of the indexer.
+ *
+ * @returns A pointer to the indexer structure.
+ * @retval NULL if an error occurred.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+inline Obi_indexer_p obi_open_indexer(OBIDMS_p dms, const char* name)
+{
+	return obi_open_avl_group(dms, name);
+}
+
+
+/**
+ * @brief Closes an indexer.
+ *
+ * @param indexer A pointer to the indexer structure to close and free.
+ *
+ * @retval 0 if the operation was successfully completed.
+ * @retval -1 if an error occurred.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+inline int obi_close_indexer(Obi_indexer_p indexer)
+{
+	return obi_close_avl_group(indexer);
+}
+
+
+/**
+ * @brief Indexes a blob in an indexer and returns the index referring to the blob.
+ *
+ * @param indexer A pointer to the indexer.
+ * @param value The blob to index.
+ *
+ * @returns The index of the blob newly added in the indexer.
+ * @retval -1 if an error occurred.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+inline index_t obi_indexer_add(Obi_indexer_p indexer, Obi_blob_p value)
+{
+	return obi_avl_group_add(indexer, value);
+}
+
+
+/**
+ * @brief Recovers a blob from an indexer.
+ *
+ * @param indexer A pointer to the indexer.
+ * @param index The index of the blob in the indexer.
+ *
+ * @returns A pointer to the blob recovered.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+inline Obi_blob_p obi_indexer_get(Obi_indexer_p indexer, index_t idx)
+{
+	return obi_avl_group_get(indexer, idx);
+}
+
+
+/**
+ * @brief Recovers the name of an indexer.
+ *
+ * @param indexer A pointer on the indexer.
+ *
+ * @returns A pointer on the name of the indexer.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+inline const char* obi_indexer_get_name(Obi_indexer_p indexer)
+{
+	return obi_avl_group_get_name(indexer);
+}
+
+
+/**
+ * @brief Builds an indexer name in the form columnname_columnversion_indexer.
+ *
+ * @warning The returned pointer has to be freed by the caller.
+ *
+ * @param column_name The name of the column associated with the indexer.
+ * @param column_version The version of the column associated with the indexer.
+ *
+ * @returns A pointer on the indexer name built.
+ * @retval NULL if an error occurred.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+char* obi_build_indexer_name(const char* column_name, obiversion_t column_version);
+
+
+#endif /* OBIBLOB_INDEXER_H_ */
+

src/obidms.c

@@ -18,12 +18,14 @@
 #include <fcntl.h>
 #include <sys/types.h>
 #include <dirent.h>
+#include <unistd.h>
 
 #include "obidms.h"
 #include "obierrno.h"
 #include "obidebug.h"
 #include "obidmscolumn.h"
-#include "private_at_functions.h"
+#include "obiblob_indexer.h"
+#include "utils.h"
 #include "obilittlebigman.h"
 
 
@@ -105,7 +107,8 @@ static char* build_directory_name(const char* dms_name)
 	char* directory_name;
 
 	// Build the database directory name
-	if (asprintf(&directory_name, "%s.obidms", dms_name) < 0)
+	directory_name = (char*) malloc((strlen(dms_name) + 8)*sizeof(char));
+	if (sprintf(directory_name, "%s.obidms", dms_name) < 0)
 	{
 		obi_set_errno(OBIDMS_MEMORY_ERROR);
 		obidebug(1, "\nProblem building an OBIDMS directory name");
@@ -130,7 +133,8 @@ static char* build_infos_file_name(const char* dms_name)
 	char* file_name;
 
 	// Build file name
-	if (asprintf(&file_name, "%s_infos", dms_name) < 0)
+	file_name = (char*) malloc((strlen(dms_name) + 7)*sizeof(char));
+	if (sprintf(file_name, "%s_infos", dms_name) < 0)
 	{
 		obi_set_errno(OBIDMS_MEMORY_ERROR);
 		obidebug(1, "\nProblem building an informations file name");
@@ -199,14 +203,14 @@ int create_dms_infos_file(int dms_file_descriptor, const char* dms_name)
  *
  **********************************************************************/
 
-int obi_dms_exists(const char* dms_name)
+int obi_dms_exists(const char* dms_path)
 {
     struct stat buffer;
 	char*       directory_name;
     int 		check_dir;
 
 	// Build and check the directory name
-    directory_name = build_directory_name(dms_name);
+    directory_name = build_directory_name(dms_path);
 	if (directory_name == NULL)
 		return -1;
 
@@ -221,14 +225,15 @@ int obi_dms_exists(const char* dms_name)
 }
 
 
-OBIDMS_p obi_create_dms(const char* dms_name)
+OBIDMS_p obi_create_dms(const char* dms_path)
 {
-	char* directory_name;
-	DIR*  dms_dir;
-	int   dms_file_descriptor;
+	char*  directory_name;
+	DIR*   dms_dir;
+	int    dms_file_descriptor;
+	size_t i, j;
 
 	// Build and check the directory name
-	directory_name = build_directory_name(dms_name);
+	directory_name = build_directory_name(dms_path);
 	if (directory_name == NULL)
 		return NULL;
 
@@ -247,7 +252,7 @@ OBIDMS_p obi_create_dms(const char* dms_name)
 		return NULL;
 	}
 
-	// Get file descriptor of DMS directory to create the AVL trees directory
+	// Get file descriptor of DMS directory to create the indexer directory
 	dms_dir = opendir(directory_name);
 	if (dms_dir == NULL)
 	{
@@ -267,30 +272,40 @@ OBIDMS_p obi_create_dms(const char* dms_name)
 		return NULL;
 	}
 
-	// Create the AVL trees directory
-	if (mkdirat(dms_file_descriptor, AVL_TREES_DIR_NAME, 00777) < 0)
+	// Create the indexer directory
+	if (mkdirat(dms_file_descriptor, INDEXER_DIR_NAME, 00777) < 0)
 	{
-		obi_set_errno(OBI_AVL_ERROR);
-		obidebug(1, "\nProblem creating an AVL trees directory");
+		obi_set_errno(OBI_INDEXER_ERROR);
+		obidebug(1, "\nProblem creating an indexer directory");
 		return NULL;
 	}
 
+	// Isolate the dms name
+	j = 0;
+	for (i=0; i<strlen(dms_path); i++)
+	{
+		if (dms_path[i] == '/')
+			j = i+1;
+		i++;
+	}
+
 	// Create the informations file
-	if (create_dms_infos_file(dms_file_descriptor, dms_name) < 0)
+	if (create_dms_infos_file(dms_file_descriptor, dms_path+j) < 0)
 		return NULL;
 
-	return obi_open_dms(dms_name);
+	return obi_open_dms(dms_path);
 }
 
 
-OBIDMS_p obi_open_dms(const char* dms_name)
+OBIDMS_p obi_open_dms(const char* dms_path)
 {
 	OBIDMS_p	dms;
-	char*	 	directory_name;
+	char*	 	relative_directory_path;
 	char* 		infos_file_name;
 	int			infos_file_descriptor;
 	bool		little_endian_dms;
 	bool 		little_endian_platform;
+	size_t		i, j;
 
 	dms = NULL;
 
@@ -304,18 +319,37 @@ OBIDMS_p obi_open_dms(const char* dms_name)
 	}
 
 	// Build and check the directory name
-	directory_name = build_directory_name(dms_name);
-	if (directory_name == NULL)
+	relative_directory_path = build_directory_name(dms_path);
+	if (relative_directory_path == NULL)
 	{
 		free(dms);
 		return NULL;
 	}
 
-	strncpy(dms->directory_name, directory_name, OBIDMS_MAX_NAME);
-	free(directory_name);
+	// Build and store the absolute path to the DMS
+	if (getcwd(dms->directory_path, MAX_PATH_LEN) == NULL)
+	{
+		obi_set_errno(OBIDMS_UNKNOWN_ERROR);
+		obidebug(1, "\nError getting the absolute path to the current working directory");
+		free(relative_directory_path);
+		return NULL;
+	}
+	strcat(dms->directory_path, "/");
+	strcat(dms->directory_path, relative_directory_path);
+	free(relative_directory_path);
+
+	// Isolate and store the dms name
+	j = 0;
+	for (i=0; i<strlen(dms_path); i++)
+	{
+		if (dms_path[i] == '/')
+			j = i+1;
+		i++;
+	}
+	strcpy(dms->dms_name, dms_path+j);
 
 	// Try to open the directory
-	dms->directory = opendir(dms->directory_name);
+	dms->directory = opendir(dms->directory_path);
 	if (dms->directory == NULL)
 	{
 		switch (errno)
@@ -352,7 +386,7 @@ OBIDMS_p obi_open_dms(const char* dms_name)
 	}
 
 	// Open informations file to check endianness
-	infos_file_name = build_infos_file_name(dms_name);
+	infos_file_name = build_infos_file_name(dms->dms_name);
 	infos_file_descriptor = openat(dms->dir_fd, infos_file_name, O_RDONLY, 0777);
 	if (infos_file_descriptor < 0)
 	{
@@ -390,24 +424,24 @@ OBIDMS_p obi_open_dms(const char* dms_name)
 
 	dms->little_endian = little_endian_dms;
 
-	// Open the AVL trees directory
-	dms->avl_directory = private_opendirat(dms->dir_fd, AVL_TREES_DIR_NAME);
-	if (dms->avl_directory == NULL)
+	// Open the indexer directory
+	dms->indexer_directory = opendir_in_dms(dms, INDEXER_DIR_NAME);
+	if (dms->indexer_directory == NULL)
 	{
 		obi_set_errno(OBIDMS_UNKNOWN_ERROR);
-		obidebug(1, "\nError opening the AVL trees directory");
+		obidebug(1, "\nError opening the indexer directory");
 		closedir(dms->directory);
 		free(dms);
 		return NULL;
 	}
 
-	// Store the AVL trees directory's file descriptor
-	dms->avl_dir_fd = dirfd(dms->avl_directory);
-	if (dms->avl_dir_fd < 0)
+	// Store the indexer directory's file descriptor
+	dms->indexer_dir_fd = dirfd(dms->indexer_directory);
+	if (dms->indexer_dir_fd < 0)
 	{
 		obi_set_errno(OBIDMS_UNKNOWN_ERROR);
-		obidebug(1, "\nError getting the file descriptor of the AVL trees directory");
-		closedir(dms->avl_directory);
+		obidebug(1, "\nError getting the file descriptor of the indexer directory");
+		closedir(dms->indexer_directory);
 		closedir(dms->directory);
 		free(dms);
 		return NULL;
@@ -415,13 +449,11 @@ OBIDMS_p obi_open_dms(const char* dms_name)
 
 	// Initialize the list of opened columns
 	dms->opened_columns = (Opened_columns_list_p) malloc(sizeof(Opened_columns_list_t));
-	(dms->opened_columns)->columns = (OBIDMS_column_p*) malloc(MAX_NB_OPENED_COLUMNS*sizeof(OBIDMS_column_p));
 	(dms->opened_columns)->nb_opened_columns = 0;
 
-	// Initialize the list of opened AVL trees
-	dms->opened_avls = (Opened_avls_list_p) malloc(sizeof(Opened_avls_list_t));
-	(dms->opened_avls)->avls = (OBIDMS_avl_p*) malloc(MAX_NB_OPENED_AVL_TREES*sizeof(OBIDMS_avl_p));
-	(dms->opened_avls)->nb_opened_avls = 0;
+	// Initialize the list of opened indexers
+	dms->opened_indexers = (Opened_indexers_list_p) malloc(sizeof(Opened_indexers_list_t));
+	(dms->opened_indexers)->nb_opened_indexers = 0;
 
 	return dms;
 }
@@ -454,7 +486,7 @@ int obi_close_dms(OBIDMS_p dms)
 		while ((dms->opened_columns)->nb_opened_columns > 0)
 			obi_close_column(*((dms->opened_columns)->columns));
 
-		// Close dms and AVL trees directories
+		// Close dms and indexer directories
 		if (closedir(dms->directory) < 0)
 		{
 			obi_set_errno(OBIDMS_MEMORY_ERROR);
@@ -462,10 +494,10 @@ int obi_close_dms(OBIDMS_p dms)
 			free(dms);
 			return -1;
 		}
-		if (closedir(dms->avl_directory) < 0)
+		if (closedir(dms->indexer_directory) < 0)
 		{
-			obi_set_errno(OBI_AVL_ERROR);
-			obidebug(1, "\nError closing an AVL trees directory");
+			obi_set_errno(OBI_INDEXER_ERROR);
+			obidebug(1, "\nError closing an indexer directory");
 			free(dms);
 			return -1;
 		}
@@ -475,3 +507,167 @@ int obi_close_dms(OBIDMS_p dms)
 	return 0;
 }
 
+
+int obi_dms_is_column_name_in_list(OBIDMS_p dms, const char* column_name)
+{
+	int i;
+	Opened_columns_list_p columns_list;
+
+	columns_list = dms->opened_columns;
+
+	for (i=0; i < (columns_list->nb_opened_columns); i++)
+	{
+		if (!strcmp(((*((columns_list->columns)+i))->header)->name, column_name))
+		{	// Found it
+			return 1;
+		}
+	}
+	return 0;
+}
+
+
+OBIDMS_column_p obi_dms_get_column_from_list(OBIDMS_p dms, const char* column_name, obiversion_t version)
+{
+	int i;
+
+	for (i=0; i < ((dms->opened_columns)->nb_opened_columns); i++)
+	{
+		if (!strcmp(((*(((dms->opened_columns)->columns)+i))->header)->name, column_name)
+				&& (((*(((dms->opened_columns)->columns)+i))->header)->version == version))
+		{	// Found the column already opened, return it
+			return *(((dms->opened_columns)->columns)+i);
+		}
+	}
+	// Didn't find the column
+	return NULL;
+}
+
+
+void obi_dms_list_column(OBIDMS_p dms, OBIDMS_column_p column)	// TODO add check if column already in list?
+{
+	*(((dms->opened_columns)->columns)+((dms->opened_columns)->nb_opened_columns)) = column;
+	((dms->opened_columns)->nb_opened_columns)++;
+}
+
+
+int obi_dms_unlist_column(OBIDMS_p dms, OBIDMS_column_p column)
+{
+	int i;
+	Opened_columns_list_p columns_list;
+
+	columns_list = dms->opened_columns;
+
+	for (i=0; i < columns_list->nb_opened_columns; i++)
+	{
+		if (!strcmp(((*((columns_list->columns)+i))->header)->name, (column->header)->name)
+				&& (((*((columns_list->columns)+i))->header)->version == (column->header)->version))
+		{	// Found the column. Rearrange list
+			(columns_list->nb_opened_columns)--;
+			(columns_list->columns)[i] = (columns_list->columns)[columns_list->nb_opened_columns];
+			return 0;
+		}
+	}
+
+	obidebug(1, "\nCould not find the column to delete from list of open columns");
+	return -1;
+}
+
+
+Obi_indexer_p obi_dms_get_indexer_from_list(OBIDMS_p dms, const char* indexer_name)
+{
+	int i;
+	Opened_indexers_list_p indexers_list;
+
+	indexers_list = dms->opened_indexers;
+
+	for (i=0; i < (indexers_list->nb_opened_indexers); i++)
+	{
+		if (!strcmp(obi_indexer_get_name((indexers_list->indexers)[i]), indexer_name))
+		{	// Found the indexer already opened, return it
+			return (indexers_list->indexers)[i];
+		}
+	}
+	// Didn't find the indexer
+	return NULL;
+}
+
+
+void obi_dms_list_indexer(OBIDMS_p dms, Obi_indexer_p indexer)  	// TODO add check if indexer already in list?
+{
+	*(((dms->opened_indexers)->indexers)+((dms->opened_indexers)->nb_opened_indexers)) = indexer;
+	((dms->opened_indexers)->nb_opened_indexers)++;
+}
+
+
+int obi_dms_unlist_indexer(OBIDMS_p dms, Obi_indexer_p indexer)
+{
+	int i;
+	Opened_indexers_list_p indexers_list;
+
+	indexers_list = dms->opened_indexers;
+
+	for (i=0; i < indexers_list->nb_opened_indexers; i++)
+	{
+		if (!strcmp(obi_indexer_get_name((indexers_list->indexers)[i]), indexer->name))
+		{	// Found the indexer. Rearrange list
+			(indexers_list->nb_opened_indexers)--;
+			(indexers_list->indexers)[i] = (indexers_list->indexers)[indexers_list->nb_opened_indexers];
+			return 0;
+		}
+	}
+
+	obidebug(1, "\nCould not find the indexer to delete from list of open indexers");
+	return -1;
+}
+
+
+char* obi_dms_get_dms_path(OBIDMS_p dms)
+{
+	char* full_path;
+
+	full_path = (char*) malloc((MAX_PATH_LEN)*sizeof(char));
+	if (full_path == NULL)
+	{
+		obi_set_errno(OBI_MALLOC_ERROR);
+		obidebug(1, "\nError allocating memory for the char* path to a file or directory");
+		return NULL;
+	}
+
+	strcpy(full_path, dms->directory_path);
+
+	return full_path;
+}
+
+
+char* obi_dms_get_full_path(OBIDMS_p dms, const char* path_name)
+{
+	char* full_path;
+
+	full_path = obi_dms_get_dms_path(dms);
+	strcat(full_path, "/");
+	strcat(full_path, path_name);
+
+	return full_path;
+}
+
+
+DIR* opendir_in_dms(OBIDMS_p dms, const char* path_name)
+{
+	char* full_path;
+	DIR* directory;
+
+	full_path = obi_dms_get_full_path(dms, path_name);
+	if (full_path == NULL)
+		return NULL;
+
+	directory = opendir(full_path);
+	if (directory == NULL)
+	{
+		obi_set_errno(OBI_UTILS_ERROR);
+		obidebug(1, "\nError opening a directory");
+	}
+
+	free(full_path);
+
+	return directory;
+}

src/obidms.h

@@ -23,34 +23,52 @@
 #include <stdbool.h>
 
 #include "obierrno.h"
+#include "obitypes.h"
 
 
-#define OBIDMS_MAX_NAME (2048) 	    		/**< The maximum length of an OBIDMS name.
-                                	 	 	 */
-#define AVL_TREES_DIR_NAME "AVL_trees"    	/**< The name of the AVL trees directory.
-                                	 	 	 */
-#define TAXONOMY_DIR_NAME "TAXONOMY"    	/**< The name of the taxonomy directory.
-                                	 	 	 */
-#define MAX_NB_OPENED_COLUMNS (100) 		/**< The maximum number of columns open at the same time.
-                                	 	 	 */
-#define MAX_NB_OPENED_AVL_TREES (100)  		/**< The maximum number of AVL trees open at the same time.
-                                	 	 	 */
+#define OBIDMS_MAX_NAME (2048) 	    			/**< The maximum length of an OBIDMS name.
+ 	 	 	 	 	 	 	 	 	 	 	 	 */
+#define INDEXER_DIR_NAME "OBIBLOB_INDEXERS"  	/**< The name of the Obiblob indexer directory.
+                                	 	 	 	 */
+#define TAXONOMY_DIR_NAME "TAXONOMY"    		/**< The name of the taxonomy directory.
+                                	 	 	 	 */
+#define MAX_NB_OPENED_COLUMNS (100) 			/**< The maximum number of columns open at the same time.
+                                	 	 	 	 */
+#define MAX_NB_OPENED_INDEXERS (1000) 			/**< The maximum number of indexers open at the same time.
+                                	 	 	 	 */
+#define MAX_PATH_LEN 4096						/**< Maximum length for the character string defining a
+ 	 	 	 	 	 	 	 	 	 	 	 	 *	 file or directory path.
+ 	 	 	 	 	 	 	 	 	 	 	 	 */
 
 
-struct OBIDMS_column;		// TODO
+struct OBIDMS_column;							/**< Declarations to avoid circular dependencies. */
+typedef struct OBIDMS_column* OBIDMS_column_p;  /**< Declarations to avoid circular dependencies. */
 
+/**
+ * @brief Structure listing the columns opened in a DMS, identified by their name and version number.
+ */
 typedef struct Opened_columns_list {
-	size_t	 		 		nb_opened_columns;
-	struct OBIDMS_column** 	columns;				// TODO allocate array size here
+	int	 		 			nb_opened_columns;	  				/**< Number of opened columns.
+	 	   	   	   	   	   	   	   	   	   	   	   	   	   	   	 */
+	OBIDMS_column_p		 	columns[MAX_NB_OPENED_COLUMNS];	  	/**< Array of pointers on the opened columns.
+ 	   	   	   	   	   	   	 	 	 	 	 	 	 	 	 	 */
 } Opened_columns_list_t, *Opened_columns_list_p;
 
 
-struct OBIDMS_avl;		// TODO
+// TODO Need to find a way to not refer to AVLs specifically
+struct OBIDMS_avl_group;								/**< Declarations to avoid circular dependencies. */
+typedef struct OBIDMS_avl_group* OBIDMS_avl_group_p;	/**< Declarations to avoid circular dependencies. */
+typedef OBIDMS_avl_group_p Obi_indexer_p;				/**< Declarations to avoid circular dependencies. */
 
-typedef struct Opened_avls_list {
-	size_t	 		 		nb_opened_avls;
-	struct OBIDMS_avl** 	avls;
-} Opened_avls_list_t, *Opened_avls_list_p;
+/**
+ * @brief Structure listing the indexers opened in a DMS, identified by their name.
+ */
+typedef struct Opened_indexers_list {
+	int		 		 		nb_opened_indexers;					/**< Number of opened indexers.
+	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 */
+	Obi_indexer_p		    indexers[MAX_NB_OPENED_INDEXERS];	/**< Array of pointers on the opened indexers.
+	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 */
+} Opened_indexers_list_t, *Opened_indexers_list_p;
 
 
 /**
@@ -60,26 +78,31 @@ typedef struct Opened_avls_list {
  * and opening of an OBITools Data Management System (DMS)
  */
 typedef struct OBIDMS {
+	char					dms_name[OBIDMS_MAX_NAME+1];		/** The name of the DMS.
+																 */
 	char					directory_name[OBIDMS_MAX_NAME+1];	/**< The name of the directory
 																 *   containing the DMS.
 																 */
+	char					directory_path[MAX_PATH_LEN];     	/**< The absolute path of the directory
+																 *   containing the DMS.
+																 */
 	DIR*					directory;							/**< A directory entry usable to
 																 *   refer and scan the database directory.
 																 */
 	int     				dir_fd;								/**< The file descriptor of the directory entry
 																 *   usable to refer and scan the database directory.
 																 */
-	DIR*					avl_directory;						/**< A directory entry usable to
-																 *   refer and scan the AVL trees directory.
+	DIR*					indexer_directory;					/**< A directory entry usable to
+																 *   refer and scan the indexer directory.
 																 */
-	int						avl_dir_fd;							/**< The file descriptor of the directory entry
-																 *   usable to refer and scan the AVL trees directory.
+	int						indexer_dir_fd;						/**< The file descriptor of the directory entry
+																 *   usable to refer and scan the indexer directory.
 																 */
 	bool					little_endian;						/**< Endianness of the database.
 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 */
 	Opened_columns_list_p 	opened_columns;						/**< List of opened columns.
 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 */
-	Opened_avls_list_p 		opened_avls;						/**< List of opened AVL trees.
+	Opened_indexers_list_p 	opened_indexers;					/**< List of opened indexers.
 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 */
 } OBIDMS_t, *OBIDMS_p;
 
@@ -87,9 +110,9 @@ typedef struct OBIDMS {
 /**
  * @brief Checks if an OBIDMS exists.
  *
- * @param dms_name A pointer to a C string containing the name of the database.
+ * @param dms_path A pointer to a C string containing the path to the database.
  *
- * @returns An integer value indicating the status of the database
+ * @returns An integer value indicating the status of the database.
  * @retval 1 if the database exists.
  * @retval 0 if the database does not exist.
  * @retval -1 if an error occurred.
@@ -97,7 +120,7 @@ typedef struct OBIDMS {
  * @since May 2015
  * @author Eric Coissac (eric.coissac@metabarcoding.org)
  */
-int obi_dms_exists(const char* dms_name);
+int obi_dms_exists(const char* dms_path);
 
 
 /**
@@ -107,9 +130,9 @@ int obi_dms_exists(const char* dms_name);
  * if a directory with this name does not already exist
  * before creating the new database.
  *
- * A directory to store AVL trees is also created.
+ * A directory to store Obiblob indexers is also created.
  *
- * @param dms_name A pointer to a C string containing the name of the database.
+ * @param dms_path A pointer to a C string containing the path to the database.
  *                 The actual directory name used to store the DMS will be
  *                 `<dms_name>.obidms`.
  *
@@ -126,7 +149,7 @@ OBIDMS_p obi_create_dms(const char* dms_name);
 /**
  * @brief Opens an existing OBITools Data Management instance (OBIDMS).
  *
- * @param dms_name A pointer to a C string containing the name of the database.
+ * @param dms_path A pointer to a C string containing the path to the database.
  *
  * @returns A pointer to an OBIDMS structure describing the opened DMS.
  * @retval NULL if an error occurred.
@@ -135,7 +158,7 @@ OBIDMS_p obi_create_dms(const char* dms_name);
  * @since May 2015
  * @author Eric Coissac (eric.coissac@metabarcoding.org)
  */
-OBIDMS_p obi_open_dms(const char* dms_name);
+OBIDMS_p obi_open_dms(const char* dms_path);
 
 
 /**
@@ -144,7 +167,7 @@ OBIDMS_p obi_open_dms(const char* dms_name);
  * If the database already exists, this function opens it, otherwise it
  * creates a new database.
  *
- * @param dms_name A pointer to a C string containing the name of the database.
+ * @param dms_path A pointer to a C string containing the path to the database.
  *
  * @returns A pointer to an OBIDMS structure describing the OBIDMS.
  * @retval NULL if an error occurred.
@@ -153,7 +176,7 @@ OBIDMS_p obi_open_dms(const char* dms_name);
  * @since May 2015
  * @author Eric Coissac (eric.coissac@metabarcoding.org)
  */
-OBIDMS_p obi_dms(const char* dms_name);
+OBIDMS_p obi_dms(const char* dms_path);
 
 
 /**
@@ -174,4 +197,155 @@ OBIDMS_p obi_dms(const char* dms_name);
 int obi_close_dms(OBIDMS_p dms);
 
 
+/**
+ * @brief Checks if a column with a given name is in the list of opened columns.
+ *
+ * @warning: Checking only the name means that the column can be of any version.
+ *
+ * @param dms The OBIDMS.
+ * @param column_name The column name that should be looked for.
+ *
+ * @returns An integer value indicating whether there is at least one column with that name
+ * 			in the list of opened columns.
+ * @retval 0 if there is at least one column with that name in the list of opened columns.
+ * @retval 1 if there is no column with that name in the list of opened columns.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+int obi_dms_is_column_name_in_list(OBIDMS_p dms, const char* column_name);
+
+
+/**
+ * @brief Returns a column identified by its name and its version number from the list of opened columns.
+ *
+ * @param dms The OBIDMS.
+ * @param column_name The column name that should be looked for.
+ * @param version The version number of the column that should be looked for.
+ *
+ * @returns A pointer on the column if it was found in the list of opened columns.
+ * @retval NULL if the column was not found in the list of opened columns.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+OBIDMS_column_p obi_dms_get_column_from_list(OBIDMS_p dms, const char* column_name, obiversion_t version);
+
+
+/**
+ * @brief Adds a column identified by its name and its version number in the list of opened columns.
+ *
+ * @warning obi_dms_get_column_from_list() should be used first to check if the column isn't already listed.
+ *
+ * @param dms The OBIDMS.
+ * @param column A pointer on the column that should be added in the list of opened columns.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+void obi_dms_list_column(OBIDMS_p dms, OBIDMS_column_p column);
+
+
+/**
+ * @brief Removes a column identified by its name and its version number from the list of opened columns.
+ *
+ * @param dms The OBIDMS.
+ * @param column A pointer on the column that should be removed from the list of opened columns.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+int obi_dms_unlist_column(OBIDMS_p dms, OBIDMS_column_p column);
+
+
+/**
+ * @brief Returns an indexer identified by its name from the list of opened indexers.
+ *
+ * @param dms The OBIDMS.
+ * @param indexer_name The indexer name that should be looked for.
+ *
+ * @returns A pointer on the indexer if it was found in the list of opened indexers.
+ * @retval NULL if the indexer was not found in the list of opened indexers.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+Obi_indexer_p obi_dms_get_indexer_from_list(OBIDMS_p dms, const char* indexer_name);
+
+
+/**
+ * @brief Adds an indexer identified by its name in the list of opened indexers.
+ *
+ * @warning obi_dms_get_indexer_from_list() should be used first to check if the indexer isn't already listed.
+ *
+ * @param dms The OBIDMS.
+ * @param indexer A pointer on the indexer that should be added in the list of opened indexers.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+void obi_dms_list_indexer(OBIDMS_p dms, Obi_indexer_p indexer);
+
+
+/**
+ * @brief Removes an indexer identified by its name from the list of opened indexers.
+ *
+ * @param dms The OBIDMS.
+ * @param column A pointer on the indexer that should be removed from the list of opened indexers.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+int obi_dms_unlist_indexer(OBIDMS_p dms, Obi_indexer_p indexer);
+
+
+/**
+ * @brief Gets the full path to the DMS.
+ *
+ * @warning The returned pointer has to be freed by the caller.
+ *
+ * @param dms The DMS.
+ *
+ * @returns A pointer to the full path.
+ * @retval NULL if an error occurred.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+char* obi_dms_get_dms_path(OBIDMS_p dms);
+
+
+/**
+ * @brief Gets the full path of a file or a directory from its
+ *        path relative to the DMS.
+ *
+ * @warning The returned pointer has to be freed by the caller.
+ *
+ * @param dms The DMS to which path_name is relative.
+ * @param path_name The path name for the file or directory, relative to the DMS.
+ *
+ * @returns A pointer to the full path.
+ * @retval NULL if an error occurred.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+char* obi_dms_get_full_path(OBIDMS_p dms, const char* path_name);
+
+
+/**
+ * @brief Opens a directory relative to the DMS.
+ *
+ * @param dms The DMS to which path_name is relative.
+ * @param path_name The path name for the directory to be opened, relative to the DMS.
+ *
+ * @returns The directory stream of the opened directory.
+ * @retval NULL if an error occurred.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+DIR* opendir_in_dms(OBIDMS_p dms, const char* path_name);
+
+
 #endif /* OBIDMS_H_ */

src/obidms_taxonomy.c

@@ -20,7 +20,7 @@
 #include "obidms.h"
 #include "obidebug.h"
 #include "obierrno.h"
-#include "private_at_functions.h"
+#include "utils.h"
 
 
 #define DEBUG_LEVEL 0	// TODO has to be defined somewhere else (cython compil flag?)
@@ -379,8 +379,9 @@ OBIDMS_taxonomy_p obi_read_taxonomy(OBIDMS_p dms, const char* taxonomy_name, boo
 
 	buffer_size = 2048;		// TODO
 
-	main_taxonomy_dir_path = get_full_path(dms->dir_fd, TAXONOMY_DIR_NAME);
-	if (asprintf(&taxonomy_path, "%s/%s/%s", main_taxonomy_dir_path, taxonomy_name, taxonomy_name) < 0)
+	main_taxonomy_dir_path = obi_dms_get_full_path(dms, TAXONOMY_DIR_NAME);
+	taxonomy_path = (char*) malloc((strlen(main_taxonomy_dir_path) + strlen(taxonomy_name) + strlen(taxonomy_name) + 3)*sizeof(char));
+	if (sprintf(taxonomy_path, "%s/%s/%s", main_taxonomy_dir_path, taxonomy_name, taxonomy_name) < 0)
 	{
 		free(main_taxonomy_dir_path);
 		obi_close_taxonomy(tax);

src/obidmscolumn.c

@@ -29,7 +29,8 @@
 #include "obierrno.h"
 #include "obidebug.h"
 #include "obilittlebigman.h"
-#include "obiavl.h"
+#include "obiblob_indexer.h"
+#include "utils.h"
 
 
 #define DEBUG_LEVEL 0	// TODO has to be defined somewhere else (cython compil flag?)
@@ -156,9 +157,19 @@ static index_t get_line_count_per_page(OBIType_t data_type, index_t nb_elements_
 static char* build_column_file_name(const char* column_name, obiversion_t version_number)
 {
 	char* file_name;
+	int version_number_length;
 
 	// Build the file name
-	if (asprintf(&file_name,"%s@%d.odc", column_name, version_number) < 0)
+	version_number_length = (version_number == 0 ? 1 : (int)(log10(version_number)+1));
+	file_name =	(char*) malloc((strlen(column_name) + version_number_length + 6)*sizeof(char));
+	if (file_name == NULL)
+	{
+		obi_set_errno(OBI_MALLOC_ERROR);
+		obidebug(1, "\nError allocating the memory for a column file name");
+		return NULL;
+	}
+
+	if (sprintf(file_name,"%s@%d.odc", column_name, version_number) < 0)
 	{
 		obi_set_errno(OBICOL_MEMORY_ERROR);
 		obidebug(1, "\nError building a column file name");
@@ -174,7 +185,15 @@ static char* build_version_file_name(const char* column_name)
 	char* file_name;
 
 	// Build the file name
-	if (asprintf(&file_name,"%s.odv", column_name) < 0)
+	file_name =	(char*) malloc((strlen(column_name) + 5)*sizeof(char));
+	if (file_name == NULL)
+	{
+		obi_set_errno(OBI_MALLOC_ERROR);
+		obidebug(1, "\nError allocating the memory for a version file name");
+		return NULL;
+	}
+
+	if (sprintf(file_name,"%s.odv", column_name) < 0)
 	{
 		obi_set_errno(OBICOL_MEMORY_ERROR);
 		obidebug(1, "\nError building a version file name");
@@ -514,14 +533,13 @@ OBIDMS_column_p obi_create_column(OBIDMS_p    dms,
 								  index_t     nb_lines,
 								  index_t     nb_elements_per_line,
 								  const char* elements_names,
-								  const char* avl_name,
+								  const char* indexer_name,
 								  const char* comments
 								 )
 {
 	OBIDMS_column_p 			new_column;
 	OBIDMS_column_directory_p	column_directory;
 	OBIDMS_column_header_p 		header;
-	OBIDMS_avl_p				avl;
 	size_t 						file_size;
 	obiversion_t 				version_number;
 	char* 						column_file_name;
@@ -531,6 +549,7 @@ OBIDMS_column_p obi_create_column(OBIDMS_p    dms,
 	index_t						minimum_line_count;
 	OBIType_t  					returned_data_type;
 	OBIType_t  					stored_data_type;
+	char*			    		final_indexer_name;
 
 	new_column = NULL;
 
@@ -550,12 +569,39 @@ OBIDMS_column_p obi_create_column(OBIDMS_p    dms,
 		obidebug(1, "\nCan't create column because of invalid data type");
 		return NULL;
 	}
-	if (((data_type == OBI_STR) || (data_type == OBI_SEQ)) && (avl_name == NULL))
+
+	// Get the column directory structure associated to the column
+	column_directory = obi_column_directory(dms, column_name);
+	if (column_directory == NULL)
 	{
-		obidebug(1, "\nCan't create column because of empty avl name");
+		obi_set_errno(OBICOLDIR_UNKNOWN_ERROR);
+		obidebug(1, "\nError opening a column directory structure");
 		return NULL;
 	}
 
+	// Get the latest version number
+	version_number = obi_get_new_version_number(column_directory, true);
+	if (version_number < 0)
+	{
+		return NULL;
+	}
+
+	// Build the indexer name if needed
+	if ((data_type == OBI_STR) || (data_type == OBI_SEQ))
+	{
+		if (strcmp(indexer_name, "") == 0)
+		{
+			final_indexer_name = obi_build_indexer_name(column_name, version_number);
+			if (final_indexer_name == NULL)
+				return NULL;
+		}
+		else
+		{
+			final_indexer_name = (char*) malloc((strlen(indexer_name)+1)*sizeof(char));
+			strcpy(final_indexer_name, indexer_name);
+		}
+	}
+
 	returned_data_type = data_type;
 	if ((data_type == OBI_STR) || (data_type == OBI_SEQ))
 	// stored data is indices referring to data stored elsewhere
@@ -602,27 +648,11 @@ OBIDMS_column_p obi_create_column(OBIDMS_p    dms,
 		return NULL;
 	}
 
-	// Get the column directory structure associated to the column
-	column_directory = obi_column_directory(dms, column_name);
-	if (column_directory == NULL)
-	{
-		obi_set_errno(OBICOLDIR_UNKNOWN_ERROR);
-		obidebug(1, "\nError opening a column directory structure");
-		return NULL;
-	}
-
 	// Calculate the size needed
 	header_size = obi_get_platform_header_size();
 	data_size = obi_array_sizeof(stored_data_type, nb_lines, nb_elements_per_line);
 	file_size = header_size + data_size;
 
-	// Get the latest version number
-	version_number = obi_get_new_version_number(column_directory, true);
-	if (version_number < 0)
-	{
-		return NULL;
-	}
-
 	// Get the column file name
 	column_file_name = build_column_file_name(column_name, version_number);
 	if (column_file_name == NULL)
@@ -720,20 +750,19 @@ OBIDMS_column_p obi_create_column(OBIDMS_p    dms,
 	if (comments != NULL)
 		strncpy(header->comments, comments, COMMENTS_MAX_LENGTH);
 
-	// If the data type is OBI_STR or OBI_SEQ, the associated obi_avl is opened or created
+	// If the data type is OBI_STR or OBI_SEQ, the associated obi_indexer is opened or created
 	if ((returned_data_type == OBI_STR) || (returned_data_type == OBI_SEQ))
 	{
-		avl = obi_avl(dms, avl_name);
-		if (avl == NULL)
+		new_column->indexer = obi_indexer(dms, final_indexer_name);
+		if (new_column->indexer == NULL)
 		{
-			obidebug(1, "\nError opening or creating the aVL tree associated with a column");
+			obidebug(1, "\nError opening or creating the indexer associated with a column");
 			munmap(new_column->header, header_size);
 			close(column_file_descriptor);
 			free(new_column);
 			return NULL;
 		}
-		new_column->avl = avl;
-		strncpy(header->avl_name, avl_name, AVL_MAX_NAME);
+		strncpy(header->indexer_name, final_indexer_name, INDEXER_MAX_NAME);
 	}
 
 	// Fill the data with NA values
@@ -742,8 +771,9 @@ OBIDMS_column_p obi_create_column(OBIDMS_p    dms,
 	close(column_file_descriptor);
 
 	// Add in the list of opened columns
-	*(((dms->opened_columns)->columns)+((dms->opened_columns)->nb_opened_columns)) = new_column;
-	((dms->opened_columns)->nb_opened_columns)++;
+	obi_dms_list_column(dms, new_column);
+
+	// Set counter to 1 // TODO Discuss counters
 	new_column->counter = 1;
 
 	return new_column;
@@ -756,11 +786,9 @@ OBIDMS_column_p obi_open_column(OBIDMS_p dms,
 {
 	OBIDMS_column_p 		    column;
 	OBIDMS_column_directory_p	column_directory;
-	OBIDMS_avl_p				avl;
 	char* 						column_file_name;
 	int 						column_file_descriptor;
 	size_t 						header_size;
-	size_t						i;
 
 	column = NULL;
 
@@ -768,7 +796,7 @@ OBIDMS_column_p obi_open_column(OBIDMS_p dms,
 	column_directory = obi_open_column_directory(dms, column_name);
 	if (column_directory == NULL)
 	{
-		//obidebug(1, "\nError opening a column directory structure");
+		obidebug(1, "\nError opening a column directory structure");
 		return NULL;
 	}
 
@@ -784,14 +812,12 @@ OBIDMS_column_p obi_open_column(OBIDMS_p dms,
 	}
 
 	// Check if the column is already in the list of opened columns
-	for (i=0; i < ((dms->opened_columns)->nb_opened_columns); i++)
+	column = obi_dms_get_column_from_list(dms, column_name, version_number);
+	// If it's found, increment its counter and return it
+	if (column != NULL)
 	{
-		if (!strcmp(((*(((dms->opened_columns)->columns)+i))->header)->name, column_name)
-				&& (((*(((dms->opened_columns)->columns)+i))->header)->version == version_number))
-		{	// Found the column already opened, increase its counter and return the column
-			((*(((dms->opened_columns)->columns)+i))->counter)++;
-			return *(((dms->opened_columns)->columns)+i);
-		}
+		(column->counter)++;
+		return column;
 	}
 
 	// Get the column file name
@@ -806,7 +832,7 @@ OBIDMS_column_p obi_open_column(OBIDMS_p dms,
 	if (column_file_descriptor < 0)
 	{
 		obi_set_errno(OBICOL_UNKNOWN_ERROR);
-		//obidebug(1, "\nError opening column file");
+		obidebug(1, "\nError opening column file");
 		free(column_file_name);
 		return NULL;
 	}
@@ -839,7 +865,7 @@ OBIDMS_column_p obi_open_column(OBIDMS_p dms,
 	column->header 			 = mmap(NULL,
 			                  	  	header_size,
 			                  	  	PROT_READ,
-			                  	  	MAP_PRIVATE,
+			                  	  	MAP_SHARED,
 			                  	  	column_file_descriptor,
 			                  	  	0
 			                 	   );
@@ -857,7 +883,7 @@ OBIDMS_column_p obi_open_column(OBIDMS_p dms,
 	column->data   			 = mmap(NULL,
 			                 	 	(column->header)->data_size,
 			                 	 	PROT_READ,
-									MAP_PRIVATE,
+									MAP_SHARED,
 			                 	 	column_file_descriptor,
 			                 	 	header_size
 			                	   );
@@ -874,26 +900,26 @@ OBIDMS_column_p obi_open_column(OBIDMS_p dms,
 
 	column->writable = false;
 
-	// If the data type is OBI_STR or OBI_SEQ, the associated AVL tree is opened
+	// If the data type is OBI_STR or OBI_SEQ, the associated indexer is opened
 	if (((column->header)->returned_data_type == OBI_STR) || ((column->header)->returned_data_type == OBI_SEQ))
 	{
-		avl = obi_avl(dms, (column->header)->avl_name);
-		if (avl == NULL)
+		column->indexer = obi_open_indexer(dms, (column->header)->indexer_name);
+		if (column->indexer == NULL)
 		{
-			obidebug(1, "\nError opening the AVL tree associated with a column");
+			obidebug(1, "\nError opening the indexer associated with a column");
 			munmap(column->header, header_size);
 			close(column_file_descriptor);
 			free(column);
 			return NULL;
 		}
-		column->avl = avl;
 	}
 
 	close(column_file_descriptor);
 
-	// Add in the list of opened columns and set column counter to 1
-	*(((dms->opened_columns)->columns)+((dms->opened_columns)->nb_opened_columns)) = column;
-	((dms->opened_columns)->nb_opened_columns)++;
+	// Add in the list of opened columns
+	obi_dms_list_column(dms, column);
+
+	// Set counter to 1
 	column->counter = 1;
 
 	return column;
@@ -939,7 +965,7 @@ OBIDMS_column_p obi_clone_column(OBIDMS_p         dms,
 								   nb_lines,
 								   nb_elements_per_line,
 								   (column_to_clone->header)->elements_names,
-								   (column_to_clone->header)->avl_name,
+								   (column_to_clone->header)->indexer_name,
 								   (column_to_clone->header)->comments
 								  );
 
@@ -962,18 +988,13 @@ OBIDMS_column_p obi_clone_column(OBIDMS_p         dms,
 	}
 	else if (clone_data && (line_selection != NULL))
 	{
-		obidebug(1, "\nCloning data from line selection\n");
 		line_size = obi_sizeof((new_column->header)->stored_data_type) * (new_column->header)->nb_elements_per_line;
-		fprintf(stderr, "\nline size = %ld\n", line_size);
 		for (i=0; i<((line_selection->header)->lines_used); i++)
 		{
 			index = *(((index_t*) (line_selection->data)) + i);
-			fprintf(stderr, "\nindex = %lld, i = %lld\n", index, i);
 			memcpy((new_column->data)+(i*line_size), (column_to_clone->data)+(index*line_size), line_size);
-			fprintf(stderr, "\nmemcpied\n");
 		}
 		(new_column->header)->lines_used = (line_selection->header)->lines_used;
-		obidebug(1, "\nCloned data from line selection\n");
 	}
 
 	// Close column_to_clone
@@ -989,50 +1010,35 @@ OBIDMS_column_p obi_clone_column(OBIDMS_p         dms,
 
 int obi_close_column(OBIDMS_column_p column)
 {
-	size_t 				  i;
-	bool   				  close_dir;
-	Opened_columns_list_p columns_list;
+	bool    close_dir;
+	int     ret_val = 0;
 
-	columns_list = (column->dms)->opened_columns;
+	// Truncate the column to the number of lines used if it's not read-only
+	if (column->writable)
+		ret_val = obi_truncate_column(column);
 
 	(column->counter)--;
 
 	if (column->counter == 0)
 	{
 		// Delete from the list of opened columns
-		for (i=0; i < (columns_list)->nb_opened_columns; i++)
-		{
-			if (!strcmp(((*((columns_list->columns)+i))->header)->name, (column->header)->name)
-					&& (((*((columns_list->columns)+i))->header)->version == (column->header)->version))
-			{	// Found the column. Rearrange list
-				(columns_list->nb_opened_columns)--;
-				(columns_list->columns)[i] = (columns_list->columns)[columns_list->nb_opened_columns];
-			}
-		}
+		if (obi_dms_unlist_column(column->dms, column) < 0)
+			ret_val = -1;
 
 		// Close column directory if it was the last column opened from that directory
-		close_dir = 1;
-		for (i=0; i < (columns_list->nb_opened_columns); i++)
-		{
-			if (!strcmp(((*((columns_list->columns)+i))->header)->name, (column->header)->name))
-			{	// Not the last column from that directory
-				close_dir = 0;
-			}
-		}
+		close_dir = obi_dms_is_column_name_in_list(column->dms, (column->header)->name);
 
-		// If the data type is OBI_STR or OBI_SEQ, the associated AVL tree is closed
+		// If the data type is OBI_STR or OBI_SEQ, the associated indexer is closed
 		if (((column->header)->returned_data_type == OBI_STR) || ((column->header)->returned_data_type == OBI_SEQ))
-		{
-			if (obi_close_avl(column->avl) < 0)
-				return -1;
-		}
+			if (obi_close_indexer(column->indexer) < 0)
+				ret_val = -1;
 
 		// Munmap data
 		if (munmap(column->data, (column->header)->data_size) < 0)
 		{
 			obi_set_errno(OBICOL_UNKNOWN_ERROR);
 			obidebug(1, "\nError munmapping column data");
-			return -1;
+			ret_val = -1;
 		}
 
 		// Munmap header
@@ -1040,20 +1046,22 @@ int obi_close_column(OBIDMS_column_p column)
 		{
 			obi_set_errno(OBICOL_UNKNOWN_ERROR);
 			obidebug(1, "\nError munmapping a column header");
-			return -1;
+			ret_val = -1;
 		}
 
-		free(column);
-
+		// Close column directory
 		if (close_dir)
-			obi_close_column_directory(column->column_directory);
+			if (obi_close_column_directory(column->column_directory) < 0)
+				ret_val = -1;
+
+		free(column);
 	}
 
-	return 0;
+	return ret_val;
 }
 
 
-int obi_truncate_column_to_lines_used(OBIDMS_column_p column)	// TODO is it necessary to unmap/remap?
+int obi_truncate_column(OBIDMS_column_p column)	// TODO is it necessary to unmap/remap?
 {
 	size_t  file_size;
 	size_t  data_size;
@@ -1228,20 +1236,6 @@ int obi_enlarge_column(OBIDMS_column_p column)
 }
 
 
-int obi_truncate_and_close_column(OBIDMS_column_p column)
-{
-	if (column->writable)	// TODO discuss
-	{
-		if (obi_truncate_column_to_lines_used(column) < 0)
-			return -1;
-	}
-	if (obi_close_column(column) < 0)
-		return -1;
-
-	return 0;
-}
-
-
 void obi_ini_to_NA_values(OBIDMS_column_p column,
 						  index_t first_line_nb,
 						  index_t nb_lines)
@@ -1280,8 +1274,18 @@ void obi_ini_to_NA_values(OBIDMS_column_p column,
 					}
 					break;
 
-	case OBI_STR:   // fallthrough
-	case OBI_SEQ:   // fallthrough
+	case OBI_STR:   for (i=start;i<end;i++)
+					{
+						*(((index_t*) (column->data)) + i) = OBIIdx_NA;
+					}
+					break;
+
+	case OBI_SEQ:   for (i=start;i<end;i++)
+					{
+						*(((index_t*) (column->data)) + i) = OBIIdx_NA;
+					}
+					break;
+
 	case OBI_IDX:   for (i=start;i<end;i++)
 					{
 						*(((index_t*) (column->data)) + i) = OBIIdx_NA;
@@ -1348,12 +1352,12 @@ OBIDMS_column_header_p obi_column_get_header_from_name(OBIDMS_p dms, const char*
 
 	// Fill the header structure
 	header = mmap(NULL,
-			 header_size,
-			 PROT_READ,
-			 MAP_SHARED,
-			 column_file_descriptor,
-			 0
-			 );
+			 	  header_size,
+				  PROT_READ,
+				  MAP_SHARED,
+				  column_file_descriptor,
+				  0
+			 	 );
 
 	if (header == MAP_FAILED)
 	{
@@ -1369,7 +1373,7 @@ OBIDMS_column_header_p obi_column_get_header_from_name(OBIDMS_p dms, const char*
 }
 
 
-int obi_unmap_header(OBIDMS_column_header_p header)
+int obi_close_header(OBIDMS_column_header_p header)
 {
 	if (munmap(header, header->header_size) < 0)
 	{
@@ -1397,7 +1401,7 @@ index_t obi_column_get_element_index_from_name(OBIDMS_column_p column, const cha
 
 	element_index = 0;
 
-	name = strtok (elements_names, ";");	// TODO not thread safe, see strtok_r maybe
+	name = strtok(elements_names, ";");	// TODO not thread safe, see strtok_r maybe
 	if (strcmp(element_name, name) == 0)
 	{
 		free(elements_names);
@@ -1407,7 +1411,7 @@ index_t obi_column_get_element_index_from_name(OBIDMS_column_p column, const cha
 
 	while (name != NULL)
 	{
-		name = strtok (NULL, ";");			// TODO not thread safe, see strtok_r maybe
+		name = strtok(NULL, ";");			// TODO not thread safe, see strtok_r maybe
 		if (strcmp(element_name, name) == 0)
 		{
 			free(elements_names);
@@ -1422,28 +1426,48 @@ index_t obi_column_get_element_index_from_name(OBIDMS_column_p column, const cha
 }
 
 
-char* obi_column_format_date(time_t date)
+int obi_column_prepare_to_set_value(OBIDMS_column_p column, index_t line_nb)
 {
-	char*       formatted_time;
-	struct tm*  tmp;
-
-	formatted_time = (char*) malloc(FORMATTED_TIME_LENGTH*sizeof(char));
-
-	tmp = localtime(&date);
-	if (tmp == NULL)
+	// Check if the column is read-only
+	if (!(column->writable))
 	{
 		obi_set_errno(OBICOL_UNKNOWN_ERROR);
-		obidebug(1, "\nError formatting a date");
-		return NULL;
+		obidebug(1, "\nError trying to set a value in a read-only column");
+    	return -1;
 	}
 
-	if (strftime(formatted_time, FORMATTED_TIME_LENGTH, "%c", tmp) == 0)
+	// Check that the line number is not greater than the maximum allowed
+	if (line_nb >= MAXIMUM_LINE_COUNT)
 	{
 		obi_set_errno(OBICOL_UNKNOWN_ERROR);
-		obidebug(1, "\nError formatting a date");
-		return NULL;
+		obidebug(1, "\nError trying to set a value at a line number greater than the maximum allowed");
+    	return -1;
 	}
 
-	return formatted_time;
+	// Check if the file needs to be enlarged
+	while ((line_nb+1) > (column->header)->line_count)
+	{
+		// Enlarge the file
+		if (obi_enlarge_column(column) < 0)
+	    	return -1;
+	}
+
+	// Update lines used
+	if ((line_nb+1) > (column->header)->lines_used)
+		(column->header)->lines_used = line_nb+1;
+
+	return 0;
+}
+
+
+int obi_column_prepare_to_get_value(OBIDMS_column_p column, index_t line_nb)
+{
+	if ((line_nb+1) > ((column->header)->line_count))
+	{
+		obi_set_errno(OBICOL_UNKNOWN_ERROR);
+		obidebug(1, "\nError trying to get a value that is beyond the current number of lines used in the column");
+		return -1;
+	}
+	return 0;
 }
 

src/obidmscolumn.h

@@ -25,25 +25,18 @@
 #include "obierrno.h"
 #include "obilittlebigman.h"
 #include "obidmscolumndir.h"
-#include "obiavl.h"
+#include "obiblob_indexer.h"
 
 
-#define ONE_IF_ZERO(x) (((x)==0)?1:(x))	/**< If x is equal to 0, x takes the value 1.
-                                	   	 */
 #define ELEMENTS_NAMES_MAX (2048)     	/**< The maximum length of the list of elements names.
                                 	     */
 #define COLUMN_GROWTH_FACTOR (2)	 	/**< The growth factor when a column is enlarged.
                                 	   	 */
 #define MAXIMUM_LINE_COUNT (1000000000) /**< The maximum line count for the data of a column. //TODO
                                 	     */
-#define FORMATTED_TIME_LENGTH (1024)    /**< The length allocated for the character string containing a formatted date.
- 	 	 	 	 	 	 	 	 	     */
 #define COMMENTS_MAX_LENGTH (2048)      /**< The maximum length for comments.
  	 	 	 	 	 	 	 	 	     */
 
-typedef int32_t obiversion_t; 	/**< Used to store the column version number
-                                 */
-
 
 /**
  * @brief OBIDMS column header structure.
@@ -78,7 +71,7 @@ typedef struct OBIDMS_column_header {
 															 */
 	char            name[OBIDMS_COLUMN_MAX_NAME+1]; 	    /**< The column name as a NULL terminated string.
 	                                             	 	 	 */
-	char            avl_name[AVL_MAX_NAME+1]; 	    		/**< If there is one, the AVL tree name as a NULL terminated string.
+	char            indexer_name[INDEXER_MAX_NAME+1]; 		/**< If there is one, the indexer name as a NULL terminated string.
 	                                             	 	 	 */
 	char 			comments[COMMENTS_MAX_LENGTH+1];		/**< Comments stored as a classical zero end C string.
 												 	 	 	 */
@@ -98,7 +91,7 @@ typedef struct OBIDMS_column {
 	 	 	 	 	 	 	 	 	 	 	 	 	 */
 	OBIDMS_column_header_p		header; 		 	/**< A pointer to the header of the column.
 	 	 	 	 	 	 	 	 	 	 	 	 	 */
-	OBIDMS_avl_p 				avl;				/**< A pointer to the AVL tree associated with the column if there is one.
+	Obi_indexer_p    			indexer;		    /**< A pointer to the blob indexer associated with the column if there is one.
 	 	 	 	 	 	 	 	 	 	 	 	 	 */
 	void*                   	data;   		 	/**< A `void` pointer to the beginning of the data.
 	                                 	 	         *
@@ -164,7 +157,7 @@ size_t obi_get_platform_header_size();
  * @brief Creates a column.
  *
  * The minimum data size allocated is one memory page, and the data is initialized to the NA value of the OBIType.
- * If there is an AVL tree associated with the column, it is opened or created if it does not already exist.
+ * If there is an indexer associated with the column, it is opened or created if it does not already exist.
  *
  * @warning If there is one element per line, elements_names should be equal to column_name.	// TODO change this condition?
  *
@@ -174,7 +167,7 @@ size_t obi_get_platform_header_size();
  * @param nb_lines The number of lines to be stored.
  * @param nb_elements_per_line The number of elements per line.								// TODO talk about default values
  * @param elements_names The names of the elements with ';' as separator.
- * @param avl_name The name of the AVL tree if there is one associated with the column.
+ * @param indexer_name The name of the indexer if there is one associated with the column.
  * @param comments Optional comments associated with the column.
  *
  * @returns A pointer on the newly created column structure.
@@ -189,7 +182,7 @@ OBIDMS_column_p obi_create_column(OBIDMS_p dms,
 								  index_t nb_lines,
 								  index_t nb_elements_per_line,
 								  const char* elements_names,
-								  const char* avl_name,
+								  const char* indexer_name,
 								  const char* comments
 								 );
 
@@ -228,7 +221,7 @@ OBIDMS_column_p obi_clone_column(OBIDMS_p dms, OBIDMS_column_p line_selection, c
 
 
 /**
- * @brief Closes a column.
+ * @brief Truncates a column to the number of lines used if it is not read-only and closes it.
  *
  * @param column A pointer on an OBIDMS column.
  *
@@ -253,7 +246,7 @@ int obi_close_column(OBIDMS_column_p column);
  * @since August 2015
  * @author Celine Mercier (celine.mercier@metabarcoding.org)
  */
-int obi_truncate_column_to_lines_used(OBIDMS_column_p column);
+int obi_truncate_column(OBIDMS_column_p column);
 
 
 /**
@@ -270,21 +263,6 @@ int obi_truncate_column_to_lines_used(OBIDMS_column_p column);
 int obi_enlarge_column(OBIDMS_column_p column);
 
 
-/**
- * @brief Truncates a column file to the number of lines used rounded to the nearest
- *        greater multiple of the page size and closes it.
- *
- * @param column A pointer on an OBIDMS column.
- *
- * @retval 0 if the operation was successfully completed.
- * @retval -1 if an error occurred.
- *
- * @since August 2015
- * @author Celine Mercier (celine.mercier@metabarcoding.org)
- */
-int obi_truncate_and_close_column(OBIDMS_column_p column);
-
-
 /*
  * @brief Sets the data in a column to the NA value of the data OBIType.
  *
@@ -328,7 +306,7 @@ OBIDMS_column_header_p obi_column_get_header_from_name(OBIDMS_p dms, const char*
  * @since October 2015
  * @author Celine Mercier (celine.mercier@metabarcoding.org)
  */
-int obi_unmap_header(OBIDMS_column_header_p header);
+int obi_close_header(OBIDMS_column_header_p header);
 
 
 /**
@@ -347,22 +325,33 @@ index_t obi_column_get_element_index_from_name(OBIDMS_column_p column, const cha
 
 
 /**
- * @brief Formats a date in a way that is easy to read.
+ * @brief Prepares a column to set a value.
  *
- * @warning The pointer returned must be freed by the caller.
+ * @param column A pointer on an OBIDMS column.
+ * @param line_nb The number of the line at which the value will be set.
  *
- * @param date A date.
+ * @retval 0 if the operation was successfully completed.
+ * @retval -1 if an error occurred.
  *
- * @returns The date formatted in a way that is easy to read.
- *
- * @since October 2015
+ * @since April 2016
  * @author Celine Mercier (celine.mercier@metabarcoding.org)
  */
-char* obi_column_format_date(time_t date);
+int obi_column_prepare_to_set_value(OBIDMS_column_p column, index_t line_nb);
 
 
-int obi_select(OBIDMS_column_p lines_column, index_t line_to_grep);
-
+/**
+ * @brief Prepares a column to recover a value.
+ *
+ * @param column A pointer on an OBIDMS column.
+ * @param line_nb The number of the line at which the value will be recovered.
+ *
+ * @retval 0 if the operation was successfully completed.
+ * @retval -1 if an error occurred.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+int obi_column_prepare_to_get_value(OBIDMS_column_p column, index_t line_nb);
 
 
 #endif /* OBIDMSCOLUMN_H_ */

src/obidmscolumn_bool.c

@@ -14,13 +14,7 @@
 #include <stdio.h>
 
 #include "obidmscolumn.h"
-#include "obiview.h"
 #include "obitypes.h"
-#include "obierrno.h"
-#include "obidebug.h"
-
-
-#define DEBUG_LEVEL 0	// TODO has to be defined somewhere else (cython compil flag?)
 
 
 /**********************************************************************
@@ -32,25 +26,8 @@
 
 int obi_column_set_obibool_with_elt_idx(OBIDMS_column_p column, index_t line_nb, index_t element_idx, obibool_t value)
 {
-	// Check that the line number is not greater than the maximum allowed
-	if (line_nb >= MAXIMUM_LINE_COUNT)
-	{
-		obi_set_errno(OBICOL_UNKNOWN_ERROR);
-		obidebug(1, "\nError trying to set a value at a line number greater than the maximum allowed");
-    	return -1;
-	}
-
-	// Check if the file needs to be enlarged
-	while ((line_nb+1) > (column->header)->line_count)
-	{
-		// Enlarge the file
-		if (obi_enlarge_column(column) < 0)
-	    	return -1;
-	}
-
-	// Update lines used
-	if ((line_nb+1) > (column->header)->lines_used)
-		(column->header)->lines_used = line_nb+1;
+	if (obi_column_prepare_to_set_value(column, line_nb) < 0)
+		return -1;
 
 	// Set the value
 	*(((obibool_t*) (column->data)) + (line_nb * ((column->header)->nb_elements_per_line)) + element_idx) = value;
@@ -59,111 +36,31 @@ int obi_column_set_obibool_with_elt_idx(OBIDMS_column_p column, index_t line_nb,
 }
 
 
-int obi_column_set_obibool_with_elt_idx_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, index_t element_idx, obibool_t value)
-{
-	// Check that the view is not read-only
-	if (view->read_only)
-	{
-		obi_set_errno(OBIVIEW_ERROR);
-		obidebug(1, "\nError trying to set a value in a column in a read-only view");
-		return -1;
-	}
-
-	if ((view->line_selection != NULL) || (!(column->writable)))
-	{
-		// Get the right line number
-		if (column->writable)
-			line_nb = *(((index_t*) ((view->line_selection)->data)) + line_nb);
-
-		column = obi_view_clone_column(view, (column->header)->name);
-		if (column == NULL)
-		{
-			obidebug(1, "\nError trying to clone a column to modify it");
-			return -1;
-		}
-	}
-
-	if ((line_nb+1) > view->line_count)
-	{
-		if (obi_view_update_lines(view, (line_nb+1)) < 0)
-			return -1;
-	}
-
-	if (obi_column_set_obibool_with_elt_idx(column, line_nb, element_idx, value) < 0)
-		return -1;
-
-	return 0;
-}
-
-
 obibool_t obi_column_get_obibool_with_elt_idx(OBIDMS_column_p column, index_t line_nb, index_t element_idx)
 {
-	if ((line_nb+1) > ((column->header)->line_count))
-	{
-		obi_set_errno(OBICOL_UNKNOWN_ERROR);
-		obidebug(1, "\nError trying to get a value that is beyond the current number of lines used in the column");
+	if (obi_column_prepare_to_get_value(column, line_nb) < 0)
 		return OBIBool_NA;
-	}
 
 	return *(((obibool_t*) (column->data)) + (line_nb * ((column->header)->nb_elements_per_line)) + element_idx);
 }
 
 
-obibool_t obi_column_get_obibool_with_elt_idx_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, index_t element_idx)
-{
-	if ((line_nb+1) > (view->line_count))
-	{
-		obi_set_errno(OBICOL_UNKNOWN_ERROR);
-		obidebug(1, "\nError trying to get a value that is beyond the current line count of the view");
-		return OBIBool_NA;
-	}
-
-	if (view->line_selection)
-		line_nb = *(((index_t*) ((view->line_selection)->data)) + line_nb);
-
-	return obi_column_get_obibool_with_elt_idx(column, line_nb, element_idx);
-}
-
-
 int obi_column_set_obibool_with_elt_name(OBIDMS_column_p column, index_t line_nb, const char* element_name, obibool_t value)
 {
-	index_t element_idx;
-	element_idx = obi_column_get_element_index_from_name(column, element_name);
+	index_t element_idx = obi_column_get_element_index_from_name(column, element_name);
 	if (element_idx == OBIIdx_NA)
 		return -1;
-	obi_column_set_obibool_with_elt_idx(column, line_nb, element_idx, value);
-	return 0;
-}
 
-
-int obi_column_set_obibool_with_elt_name_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, const char* element_name, obibool_t value)
-{
-	index_t element_idx;
-	element_idx = obi_column_get_element_index_from_name(column, element_name);
-	if (element_idx == OBIIdx_NA)
-		return -1;
-	obi_column_set_obibool_with_elt_idx_in_view(view, column, line_nb, element_idx, value);
-	return 0;
+	return obi_column_set_obibool_with_elt_idx(column, line_nb, element_idx, value);
 }
 
 
 obibool_t obi_column_get_obibool_with_elt_name(OBIDMS_column_p column, index_t line_nb, const char* element_name)
 {
-	index_t element_idx;
-
-	element_idx = obi_column_get_element_index_from_name(column, element_name);
+	index_t element_idx = obi_column_get_element_index_from_name(column, element_name);
 	if (element_idx == OBIIdx_NA)
 		return OBIBool_NA;
+
 	return obi_column_get_obibool_with_elt_idx(column, line_nb, element_idx);
 }
 
-
-obibool_t obi_column_get_obibool_with_elt_name_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, const char* element_name)
-{
-	index_t element_idx;
-
-	element_idx = obi_column_get_element_index_from_name(column, element_name);
-	if (element_idx == OBIIdx_NA)
-		return OBIBool_NA;
-	return obi_column_get_obibool_with_elt_idx_in_view(view, column, line_nb, element_idx);
-}

src/obidmscolumn_bool.h

@@ -19,7 +19,6 @@
 
 #include "obidmscolumn.h"
 #include "obitypes.h"
-#include "obiview.h"
 
 
 /**
@@ -96,79 +95,5 @@ int obi_column_set_obibool_with_elt_name(OBIDMS_column_p column, index_t line_nb
 obibool_t obi_column_get_obibool_with_elt_name(OBIDMS_column_p column, index_t line_nb, const char* element_name);
 
 
-/**
- * @brief Sets a value in an OBIDMS column containing data with the type OBI_BOOL, using the index of the element in the line.
- *
- * @warning Pointers returned by obi_open_column() don't allow writing.
- *
- * @param column A pointer as returned by obi_create_column() or obi_clone_column().
- * @param line_nb The number of the line where the value should be set.
- * @param element_idx The index of the element that should be set in the line.
- * @param value The value that should be set.
- *
- * @returns An integer value indicating the success of the operation.
- * @retval 0 on success.
- * @retval -1 if an error occurred.
- *
- * @since February 2016
- * @author Celine Mercier (celine.mercier@metabarcoding.org)
- */
-int obi_column_set_obibool_with_elt_idx_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, index_t element_idx, obibool_t value);
-
-
-/**
- * @brief Recovers a value in an OBIDMS column containing data with the type OBI_BOOL.
- *
- * @param column A pointer as returned by obi_create_column().
- * @param line_nb The number of the line where the value should be recovered.
- * @param element_idx The index of the element that should be recovered in the line.
- *
- * @returns The recovered value.
- * @retval OBIBool_NA the NA value of the type if an error occurred and obi_errno is set.
- *
- * @since February 2016
- * @author Celine Mercier (celine.mercier@metabarcoding.org)
- */
-obibool_t obi_column_get_obibool_with_elt_idx_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, index_t element_idx);
-
-
-/**
- * @brief Sets a value in an OBIDMS column containing data with the type OBI_BOOL,
- *        using the name of the element in the line.
- *
- * @warning Pointers returned by obi_open_column() don't allow writing.
- *
- * @param column A pointer as returned by obi_create_column() or obi_clone_column().
- * @param line_nb The number of the line where the value should be set.
- * @param element_name The name of the element that should be set in the line.
- * @param value The value that should be set.
- *
- * @returns An integer value indicating the success of the operation.
- * @retval 0 on success.
- * @retval -1 if an error occurred.
- *
- * @since February 2016
- * @author Celine Mercier (celine.mercier@metabarcoding.org)
- */
-int obi_column_set_obibool_with_elt_name_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, const char* element_name, obibool_t value);
-
-
-/**
- * @brief Recovers a value in an OBIDMS column containing data with the type OBI_BOOL,
- *        using the name of the element in the line.
- *
- * @param column A pointer as returned by obi_create_column() or obi_clone_column().
- * @param line_nb The number of the line where the value should be recovered.
- * @param element_name The name of the element that should be recovered in the line.
- *
- * @returns The recovered value.
- * @retval OBIBool_NA the NA value of the type if an error occurred and obi_errno is set.
- *
- * @since February 2016
- * @author Celine Mercier (celine.mercier@metabarcoding.org)
- */
-obibool_t obi_column_get_obibool_with_elt_name_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, const char* element_name);
-
-
 #endif /* OBIDMSCOLUMN_BOOL_H_ */
 

src/obidmscolumn_char.c

@@ -14,13 +14,7 @@
 #include <stdio.h>
 
 #include "obidmscolumn.h"
-#include "obiview.h"
 #include "obitypes.h"
-#include "obierrno.h"
-#include "obidebug.h"
-
-
-#define DEBUG_LEVEL 0	// TODO has to be defined somewhere else (cython compil flag?)
 
 
 /**********************************************************************
@@ -32,25 +26,8 @@
 
 int obi_column_set_obichar_with_elt_idx(OBIDMS_column_p column, index_t line_nb, index_t element_idx, obichar_t value)
 {
-	// Check that the line number is not greater than the maximum allowed
-	if (line_nb >= MAXIMUM_LINE_COUNT)
-	{
-		obi_set_errno(OBICOL_UNKNOWN_ERROR);
-		obidebug(1, "\nError trying to set a value at a line number greater than the maximum allowed");
-    	return -1;
-	}
-
-	// Check if the file needs to be enlarged
-	while ((line_nb+1) > (column->header)->line_count)
-	{
-		// Enlarge the file
-		if (obi_enlarge_column(column) < 0)
-	    	return -1;
-	}
-
-	// Update lines used
-	if ((line_nb+1) > (column->header)->lines_used)
-		(column->header)->lines_used = line_nb+1;
+	if (obi_column_prepare_to_set_value(column, line_nb) < 0)
+		return -1;
 
 	// Set the value
 	*(((obichar_t*) (column->data)) + (line_nb * ((column->header)->nb_elements_per_line)) + element_idx) = value;
@@ -59,111 +36,30 @@ int obi_column_set_obichar_with_elt_idx(OBIDMS_column_p column, index_t line_nb,
 }
 
 
-int obi_column_set_obichar_with_elt_idx_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, index_t element_idx, obichar_t value)
-{
-	// Check that the view is not read-only
-	if (view->read_only)
-	{
-		obi_set_errno(OBIVIEW_ERROR);
-		obidebug(1, "\nError trying to set a value in a column in a read-only view");
-		return -1;
-	}
-
-	if ((view->line_selection != NULL) || (!(column->writable)))
-	{
-		// Get the right line number
-		if (column->writable)
-			line_nb = *(((index_t*) ((view->line_selection)->data)) + line_nb);
-
-		column = obi_view_clone_column(view, (column->header)->name);
-		if (column == NULL)
-		{
-			obidebug(1, "\nError trying to clone a column to modify it");
-			return -1;
-		}
-	}
-
-	if ((line_nb+1) > view->line_count)
-	{
-		if (obi_view_update_lines(view, (line_nb+1)) < 0)
-			return -1;
-	}
-
-	if (obi_column_set_obichar_with_elt_idx(column, line_nb, element_idx, value) < 0)
-		return -1;
-
-	return 0;
-}
-
-
 obichar_t obi_column_get_obichar_with_elt_idx(OBIDMS_column_p column, index_t line_nb, index_t element_idx)
 {
-	if ((line_nb+1) > ((column->header)->line_count))
-	{
-		obi_set_errno(OBICOL_UNKNOWN_ERROR);
-		obidebug(1, "\nError trying to get a value that is beyond the current number of lines used in the column");
+	if (obi_column_prepare_to_get_value(column, line_nb) < 0)
 		return OBIChar_NA;
-	}
 
 	return *(((obichar_t*) (column->data)) + (line_nb * ((column->header)->nb_elements_per_line)) + element_idx);
 }
 
 
-obichar_t obi_column_get_obichar_with_elt_idx_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, index_t element_idx)
-{
-	if ((line_nb+1) > (view->line_count))
-	{
-		obi_set_errno(OBICOL_UNKNOWN_ERROR);
-		obidebug(1, "\nError trying to get a value that is beyond the current line count of the view");
-		return OBIChar_NA;
-	}
-
-	if (view->line_selection)
-		line_nb = *(((index_t*) ((view->line_selection)->data)) + line_nb);
-
-	return obi_column_get_obichar_with_elt_idx(column, line_nb, element_idx);
-}
-
-
 int obi_column_set_obichar_with_elt_name(OBIDMS_column_p column, index_t line_nb, const char* element_name, obichar_t value)
 {
-	index_t element_idx;
-	element_idx = obi_column_get_element_index_from_name(column, element_name);
+	index_t element_idx = obi_column_get_element_index_from_name(column, element_name);
 	if (element_idx == OBIIdx_NA)
 		return -1;
-	obi_column_set_obichar_with_elt_idx(column, line_nb, element_idx, value);
-	return 0;
-}
 
-
-int obi_column_set_obichar_with_elt_name_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, const char* element_name, obichar_t value)
-{
-	index_t element_idx;
-	element_idx = obi_column_get_element_index_from_name(column, element_name);
-	if (element_idx == OBIIdx_NA)
-		return -1;
-	obi_column_set_obichar_with_elt_idx_in_view(view, column, line_nb, element_idx, value);
-	return 0;
+	return obi_column_set_obichar_with_elt_idx(column, line_nb, element_idx, value);
 }
 
 
 obichar_t obi_column_get_obichar_with_elt_name(OBIDMS_column_p column, index_t line_nb, const char* element_name)
 {
-	index_t element_idx;
-
-	element_idx = obi_column_get_element_index_from_name(column, element_name);
+	index_t element_idx = obi_column_get_element_index_from_name(column, element_name);
 	if (element_idx == OBIIdx_NA)
 		return OBIChar_NA;
+
 	return obi_column_get_obichar_with_elt_idx(column, line_nb, element_idx);
 }
-
-
-obichar_t obi_column_get_obichar_with_elt_name_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, const char* element_name)
-{
-	index_t element_idx;
-
-	element_idx = obi_column_get_element_index_from_name(column, element_name);
-	if (element_idx == OBIIdx_NA)
-		return OBIChar_NA;
-	return obi_column_get_obichar_with_elt_idx_in_view(view, column, line_nb, element_idx);
-}

src/obidmscolumn_char.h

@@ -19,7 +19,6 @@
 
 #include "obidmscolumn.h"
 #include "obitypes.h"
-#include "obiview.h"
 
 
 /**
@@ -96,79 +95,5 @@ int obi_column_set_obichar_with_elt_name(OBIDMS_column_p column, index_t line_nb
 obichar_t obi_column_get_obichar_with_elt_name(OBIDMS_column_p column, index_t line_nb, const char* element_name);
 
 
-/**
- * @brief Sets a value in an OBIDMS column containing data with the type OBI_CHAR, using the index of the element in the line.
- *
- * @warning Pointers returned by obi_open_column() don't allow writing.
- *
- * @param column A pointer as returned by obi_create_column() or obi_clone_column().
- * @param line_nb The number of the line where the value should be set.
- * @param element_idx The index of the element that should be set in the line.
- * @param value The value that should be set.
- *
- * @returns An integer value indicating the success of the operation.
- * @retval 0 on success.
- * @retval -1 if an error occurred.
- *
- * @since February 2016
- * @author Celine Mercier (celine.mercier@metabarcoding.org)
- */
-int obi_column_set_obichar_with_elt_idx_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, index_t element_idx, obichar_t value);
-
-
-/**
- * @brief Recovers a value in an OBIDMS column containing data with the type OBI_CHAR.
- *
- * @param column A pointer as returned by obi_create_column().
- * @param line_nb The number of the line where the value should be recovered.
- * @param element_idx The index of the element that should be recovered in the line.
- *
- * @returns The recovered value.
- * @retval OBIChar_NA the NA value of the type if an error occurred and obi_errno is set.
- *
- * @since February 2016
- * @author Celine Mercier (celine.mercier@metabarcoding.org)
- */
-obichar_t obi_column_get_obichar_with_elt_idx_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, index_t element_idx);
-
-
-/**
- * @brief Sets a value in an OBIDMS column containing data with the type OBI_CHAR,
- *        using the name of the element in the line.
- *
- * @warning Pointers returned by obi_open_column() don't allow writing.
- *
- * @param column A pointer as returned by obi_create_column() or obi_clone_column().
- * @param line_nb The number of the line where the value should be set.
- * @param element_name The name of the element that should be set in the line.
- * @param value The value that should be set.
- *
- * @returns An integer value indicating the success of the operation.
- * @retval 0 on success.
- * @retval -1 if an error occurred.
- *
- * @since February 2016
- * @author Celine Mercier (celine.mercier@metabarcoding.org)
- */
-int obi_column_set_obichar_with_elt_name_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, const char* element_name, obichar_t value);
-
-
-/**
- * @brief Recovers a value in an OBIDMS column containing data with the type OBI_CHAR,
- *        using the name of the element in the line.
- *
- * @param column A pointer as returned by obi_create_column() or obi_clone_column().
- * @param line_nb The number of the line where the value should be recovered.
- * @param element_name The name of the element that should be recovered in the line.
- *
- * @returns The recovered value.
- * @retval OBIChar_NA the NA value of the type if an error occurred and obi_errno is set.
- *
- * @since February 2016
- * @author Celine Mercier (celine.mercier@metabarcoding.org)
- */
-obichar_t obi_column_get_obichar_with_elt_name_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, const char* element_name);
-
-
 #endif /* OBIDMSCOLUMN_CHAR_H_ */
 

src/obidmscolumn_float.c

@@ -14,13 +14,7 @@
 #include <stdio.h>
 
 #include "obidmscolumn.h"
-#include "obiview.h"
 #include "obitypes.h"
-#include "obierrno.h"
-#include "obidebug.h"
-
-
-#define DEBUG_LEVEL 0	// TODO has to be defined somewhere else (cython compil flag?)
 
 
 /**********************************************************************
@@ -31,25 +25,8 @@
 
 int obi_column_set_obifloat_with_elt_idx(OBIDMS_column_p column, index_t line_nb, index_t element_idx, obifloat_t value)
 {
-	// Check that the line number is not greater than the maximum allowed
-	if (line_nb >= MAXIMUM_LINE_COUNT)
-	{
-		obi_set_errno(OBICOL_UNKNOWN_ERROR);
-		obidebug(1, "\nError trying to set a value at a line number greater than the maximum allowed");
-    	return -1;
-	}
-
-	// Check if the file needs to be enlarged
-	while ((line_nb+1) > (column->header)->line_count)
-	{
-		// Enlarge the file
-		if (obi_enlarge_column(column) < 0)
-	    	return -1;
-	}
-
-	// Update lines used
-	if ((line_nb+1) > (column->header)->lines_used)
-		(column->header)->lines_used = line_nb+1;
+	if (obi_column_prepare_to_set_value(column, line_nb) < 0)
+		return -1;
 
 	// Set the value
 	*(((obifloat_t*) (column->data)) + (line_nb * ((column->header)->nb_elements_per_line)) + element_idx) = value;
@@ -58,111 +35,55 @@ int obi_column_set_obifloat_with_elt_idx(OBIDMS_column_p column, index_t line_nb
 }
 
 
-int obi_column_set_obifloat_with_elt_idx_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, index_t element_idx, obifloat_t value)
-{
-	// Check that the view is not read-only
-	if (view->read_only)
-	{
-		obi_set_errno(OBIVIEW_ERROR);
-		obidebug(1, "\nError trying to set a value in a column in a read-only view");
-		return -1;
-	}
-
-	if ((view->line_selection != NULL) || (!(column->writable)))
-	{
-		// Get the right line number
-		if (column->writable)
-			line_nb = *(((index_t*) ((view->line_selection)->data)) + line_nb);
-
-		column = obi_view_clone_column(view, (column->header)->name);
-		if (column == NULL)
-		{
-			obidebug(1, "\nError trying to clone a column to modify it");
-			return -1;
-		}
-	}
-
-	if ((line_nb+1) > view->line_count)
-	{
-		if (obi_view_update_lines(view, (line_nb+1)) < 0)
-			return -1;
-	}
-
-	if (obi_column_set_obifloat_with_elt_idx(column, line_nb, element_idx, value) < 0)
-		return -1;
-
-	return 0;
-}
-
-
 obifloat_t obi_column_get_obifloat_with_elt_idx(OBIDMS_column_p column, index_t line_nb, index_t element_idx)
 {
-	if ((line_nb+1) > ((column->header)->line_count))
-	{
-		obi_set_errno(OBICOL_UNKNOWN_ERROR);
-		obidebug(1, "\nError trying to get a value that is beyond the current number of lines used in the column");
+	if (obi_column_prepare_to_get_value(column, line_nb) < 0)
 		return OBIFloat_NA;
-	}
 
 	return *(((obifloat_t*) (column->data)) + (line_nb * ((column->header)->nb_elements_per_line)) + element_idx);
 }
 
 
-obifloat_t obi_column_get_obifloat_with_elt_idx_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, index_t element_idx)
-{
-	if ((line_nb+1) > (view->line_count))
-	{
-		obi_set_errno(OBICOL_UNKNOWN_ERROR);
-		obidebug(1, "\nError trying to get a value that is beyond the current line count of the view");
-		return OBIFloat_NA;
-	}
-
-	if (view->line_selection)
-		line_nb = *(((index_t*) ((view->line_selection)->data)) + line_nb);
-
-	return obi_column_get_obifloat_with_elt_idx(column, line_nb, element_idx);
-}
-
-
 int obi_column_set_obifloat_with_elt_name(OBIDMS_column_p column, index_t line_nb, const char* element_name, obifloat_t value)
 {
-	index_t element_idx;
-	element_idx = obi_column_get_element_index_from_name(column, element_name);
+	index_t element_idx = obi_column_get_element_index_from_name(column, element_name);
 	if (element_idx == OBIIdx_NA)
 		return -1;
-	obi_column_set_obifloat_with_elt_idx(column, line_nb, element_idx, value);
-	return 0;
-}
 
-
-int obi_column_set_obifloat_with_elt_name_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, const char* element_name, obifloat_t value)
-{
-	index_t element_idx;
-	element_idx = obi_column_get_element_index_from_name(column, element_name);
-	if (element_idx == OBIIdx_NA)
-		return -1;
-	obi_column_set_obifloat_with_elt_idx_in_view(view, column, line_nb, element_idx, value);
-	return 0;
+	return obi_column_set_obifloat_with_elt_idx(column, line_nb, element_idx, value);
 }
 
 
 obifloat_t obi_column_get_obifloat_with_elt_name(OBIDMS_column_p column, index_t line_nb, const char* element_name)
 {
-	index_t element_idx;
-
-	element_idx = obi_column_get_element_index_from_name(column, element_name);
+	index_t element_idx = obi_column_get_element_index_from_name(column, element_name);
 	if (element_idx == OBIIdx_NA)
 		return OBIFloat_NA;
+
 	return obi_column_get_obifloat_with_elt_idx(column, line_nb, element_idx);
 }
 
 
-obifloat_t obi_column_get_obifloat_with_elt_name_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, const char* element_name)
+small_bool_t* obi_float_get_bool_list_from_compare(OBIDMS_column_p column, obifloat_t value, OBIDMS_column_p line_selection, Comp_type_t comp)	// TODO openMP, get index or bool list
 {
-	index_t element_idx;
+	small_bool_t* bool_vector;
+	index_t i;
+	index_t nb_elements;
 
-	element_idx = obi_column_get_element_index_from_name(column, element_name);
-	if (element_idx == OBIIdx_NA)
-		return OBIFloat_NA;
-	return obi_column_get_obifloat_with_elt_idx_in_view(view, column, line_nb, element_idx);
+	nb_elements = (column->header)->lines_used * (column->header)->nb_elements_per_line;	// TODO how are vector lines handled?
+	bool_vector = (small_bool_t*) malloc(nb_elements * sizeof(small_bool_t));
+
+	// TODO check if line_selection == NULL
+	// Check and case comps
+
+	if (comp == GREATER_THAN)
+	{
+		for (i=0; i<nb_elements; i++)
+		{
+			bool_vector[i] = (*(((obifloat_t*) (column->data)) + i)) > value;	// TODO get through function, depends on vector lines problem
+		}
+	}
+
+	return bool_vector;
 }
+

src/obidmscolumn_float.h

@@ -19,7 +19,6 @@
 
 #include "obidmscolumn.h"
 #include "obitypes.h"
-#include "obiview.h"
 
 
 /**
@@ -96,78 +95,7 @@ int obi_column_set_obifloat_with_elt_name(OBIDMS_column_p column, index_t line_n
 obifloat_t obi_column_get_obifloat_with_elt_name(OBIDMS_column_p column, index_t line_nb, const char* element_name);
 
 
-/**
- * @brief Sets a value in an OBIDMS column containing data with the type OBI_FLOAT, using the index of the element in the line.
- *
- * @warning Pointers returned by obi_open_column() don't allow writing.
- *
- * @param column A pointer as returned by obi_create_column() or obi_clone_column().
- * @param line_nb The number of the line where the value should be set.
- * @param element_idx The index of the element that should be set in the line.
- * @param value The value that should be set.
- *
- * @returns An integer value indicating the success of the operation.
- * @retval 0 on success.
- * @retval -1 if an error occurred.
- *
- * @since February 2016
- * @author Celine Mercier (celine.mercier@metabarcoding.org)
- */
-int obi_column_set_obifloat_with_elt_idx_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, index_t element_idx, obifloat_t value);
-
-
-/**
- * @brief Recovers a value in an OBIDMS column containing data with the type OBI_FLOAT.
- *
- * @param column A pointer as returned by obi_create_column().
- * @param line_nb The number of the line where the value should be recovered.
- * @param element_idx The index of the element that should be recovered in the line.
- *
- * @returns The recovered value.
- * @retval OBIFloat_NA the NA value of the type if an error occurred and obi_errno is set.
- *
- * @since February 2016
- * @author Celine Mercier (celine.mercier@metabarcoding.org)
- */
-obifloat_t obi_column_get_obifloat_with_elt_idx_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, index_t element_idx);
-
-
-/**
- * @brief Sets a value in an OBIDMS column containing data with the type OBI_FLOAT,
- *        using the name of the element in the line.
- *
- * @warning Pointers returned by obi_open_column() don't allow writing.
- *
- * @param column A pointer as returned by obi_create_column() or obi_clone_column().
- * @param line_nb The number of the line where the value should be set.
- * @param element_name The name of the element that should be set in the line.
- * @param value The value that should be set.
- *
- * @returns An integer value indicating the success of the operation.
- * @retval 0 on success.
- * @retval -1 if an error occurred.
- *
- * @since February 2016
- * @author Celine Mercier (celine.mercier@metabarcoding.org)
- */
-int obi_column_set_obifloat_with_elt_name_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, const char* element_name, obifloat_t value);
-
-
-/**
- * @brief Recovers a value in an OBIDMS column containing data with the type OBI_FLOAT,
- *        using the name of the element in the line.
- *
- * @param column A pointer as returned by obi_create_column() or obi_clone_column().
- * @param line_nb The number of the line where the value should be recovered.
- * @param element_name The name of the element that should be recovered in the line.
- *
- * @returns The recovered value.
- * @retval OBIFloat_NA the NA value of the type if an error occurred and obi_errno is set.
- *
- * @since February 2016
- * @author Celine Mercier (celine.mercier@metabarcoding.org)
- */
-obifloat_t obi_column_get_obifloat_with_elt_name_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, const char* element_name);
+small_bool_t* obi_float_get_bool_list_from_compare(OBIDMS_column_p column, obifloat_t value, OBIDMS_column_p line_selection, Comp_type_t comp);
 
 
 #endif /* OBIDMSCOLUMN_FLOAT_H_ */

src/obidmscolumn_idx.c

@@ -15,11 +15,6 @@
 
 #include "obidmscolumn.h"
 #include "obitypes.h"
-#include "obierrno.h"
-#include "obidebug.h"
-
-
-#define DEBUG_LEVEL 0	// TODO has to be defined somewhere else (cython compil flag?)
 
 
 /**********************************************************************
@@ -31,25 +26,8 @@
 
 int obi_column_set_index(OBIDMS_column_p column, index_t line_nb, index_t value)
 {
-	// Check that the line number is not greater than the maximum allowed
-	if (line_nb >= MAXIMUM_LINE_COUNT)
-	{
-		obi_set_errno(OBICOL_UNKNOWN_ERROR);
-		obidebug(1, "\nError trying to set a value at a line number greater than the maximum allowed");
-    	return -1;
-	}
-
-	// Check if the file needs to be enlarged
-	while ((line_nb+1) > (column->header)->line_count)
-	{
-		// Enlarge the file
-		if (obi_enlarge_column(column) < 0)
-	    	return -1;
-	}
-
-	// Update lines used
-	if ((line_nb+1) > (column->header)->lines_used)
-		(column->header)->lines_used = line_nb+1;
+	if (obi_column_prepare_to_set_value(column, line_nb) < 0)
+		return -1;
 
 	// Set the value
 	*(((index_t*) (column->data)) + line_nb) = value;
@@ -60,12 +38,8 @@ int obi_column_set_index(OBIDMS_column_p column, index_t line_nb, index_t value)
 
 index_t obi_column_get_index(OBIDMS_column_p column, index_t line_nb)
 {
-	if ((line_nb+1) > ((column->header)->line_count))
-	{
-		obi_set_errno(OBICOL_UNKNOWN_ERROR);
-		obidebug(1, "\nError trying to get a value that is beyond the current number of lines used in the column");
+	if (obi_column_prepare_to_get_value(column, line_nb) < 0)
 		return OBIIdx_NA;
-	}
 
 	return *(((index_t*) (column->data)) + line_nb);
 }

src/obidmscolumn_idx.h

@@ -7,6 +7,9 @@
  * @author Celine Mercier
  * @date February 14th 2016
  * @brief Header file for the functions handling OBIColumns containing data with the OBIType OBI_IDX.
+ *
+ * Note: OBI_IDX columns contain indices referring to data stored elsewhere
+ *		 (for example lines in other columns) and contain only one element (index) per line.
  */
 
 
@@ -21,8 +24,43 @@
 #include "obitypes.h"
 
 
+/**
+ * @brief Sets a value in an OBIDMS column containing data with the type OBI_IDX.
+ *
+ *	Note: OBI_IDX columns contain indices referring to data stored elsewhere
+ *		  (for example lines in other columns) and contain only one element (index) per line.
+ *
+ * @warning Pointers returned by obi_open_column() don't allow writing.
+ *
+ * @param column A pointer as returned by obi_create_column() or obi_clone_column().
+ * @param line_nb The number of the line where the value should be set.
+ * @param value The value that should be set.
+ *
+ * @returns An integer value indicating the success of the operation.
+ * @retval 0 on success.
+ * @retval -1 if an error occurred.
+ *
+ * @since February 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
 int obi_column_set_index(OBIDMS_column_p column, index_t line_nb, index_t value);
 
+
+/**
+ * @brief Recovers a value in an OBIDMS column containing data with the type OBI_IDX.
+ *
+ *	Note: OBI_IDX columns contain indices referring to data stored elsewhere
+ *		  (for example lines in other columns) and contain only one element (index) per line.
+ *
+ * @param column A pointer as returned by obi_create_column() or obi_clone_column().
+ * @param line_nb The number of the line where the value should be recovered.
+ *
+ * @returns The recovered value.
+ * @retval OBIIdx_NA the NA value of the type if an error occurred and obi_errno is set.
+ *
+ * @since February 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
 index_t obi_column_get_index(OBIDMS_column_p column, index_t line_nb);
 
 

src/obidmscolumn_int.c

@@ -14,13 +14,7 @@
 #include <stdio.h>
 
 #include "obidmscolumn.h"
-#include "obiview.h"
 #include "obitypes.h"
-#include "obierrno.h"
-#include "obidebug.h"
-
-
-#define DEBUG_LEVEL 0	// TODO has to be defined somewhere else (cython compil flag?)
 
 
 /**********************************************************************
@@ -29,28 +23,10 @@
  *
  **********************************************************************/
 
-
 int obi_column_set_obiint_with_elt_idx(OBIDMS_column_p column, index_t line_nb, index_t element_idx, obiint_t value)
 {
-	// Check that the line number is not greater than the maximum allowed
-	if (line_nb >= MAXIMUM_LINE_COUNT)
-	{
-		obi_set_errno(OBICOL_UNKNOWN_ERROR);
-		obidebug(1, "\nError trying to set a value at a line number greater than the maximum allowed");
-    	return -1;
-	}
-
-	// Check if the file needs to be enlarged
-	while ((line_nb+1) > (column->header)->line_count)
-	{
-		// Enlarge the file
-		if (obi_enlarge_column(column) < 0)
-	    	return -1;
-	}
-
-	// Update lines used
-	if ((line_nb+1) > (column->header)->lines_used)
-		(column->header)->lines_used = line_nb+1;
+	if (obi_column_prepare_to_set_value(column, line_nb) < 0)
+		return -1;
 
 	// Set the value
 	*(((obiint_t*) (column->data)) + (line_nb * ((column->header)->nb_elements_per_line)) + element_idx) = value;
@@ -59,112 +35,31 @@ int obi_column_set_obiint_with_elt_idx(OBIDMS_column_p column, index_t line_nb,
 }
 
 
-int obi_column_set_obiint_with_elt_idx_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, index_t element_idx, obiint_t value)
-{
-	// Check that the view is not read-only
-	if (view->read_only)
-	{
-		obi_set_errno(OBIVIEW_ERROR);
-		obidebug(1, "\nError trying to set a value in a column in a read-only view");
-		return -1;
-	}
-
-	if ((view->line_selection != NULL) || (!(column->writable)))
-	{
-		// Get the right line number
-		if (column->writable)
-			line_nb = *(((index_t*) ((view->line_selection)->data)) + line_nb);
-
-		column = obi_view_clone_column(view, (column->header)->name);
-		if (column == NULL)
-		{
-			obidebug(1, "\nError trying to clone a column to modify it");
-			return -1;
-		}
-	}
-
-	if ((line_nb+1) > view->line_count)
-	{
-		if (obi_view_update_lines(view, (line_nb+1)) < 0)
-			return -1;
-	}
-
-	if (obi_column_set_obiint_with_elt_idx(column, line_nb, element_idx, value) < 0)
-		return -1;
-
-	return 0;
-}
-
-
 obiint_t obi_column_get_obiint_with_elt_idx(OBIDMS_column_p column, index_t line_nb, index_t element_idx)
 {
-	if ((line_nb+1) > ((column->header)->line_count))
-	{
-		obi_set_errno(OBICOL_UNKNOWN_ERROR);
-		obidebug(1, "\nError trying to get a value that is beyond the current number of lines used in the column");
+	if (obi_column_prepare_to_get_value(column, line_nb) < 0)
 		return OBIInt_NA;
-	}
 
 	return *(((obiint_t*) (column->data)) + (line_nb * ((column->header)->nb_elements_per_line)) + element_idx);
 }
 
 
-obiint_t obi_column_get_obiint_with_elt_idx_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, index_t element_idx)
-{
-	if ((line_nb+1) > (view->line_count))
-	{
-		obi_set_errno(OBICOL_UNKNOWN_ERROR);
-		obidebug(1, "\nError trying to get a value that is beyond the current line count of the view");
-		return OBIInt_NA;
-	}
-
-	if (view->line_selection)
-		line_nb = *(((index_t*) ((view->line_selection)->data)) + line_nb);
-
-	return obi_column_get_obiint_with_elt_idx(column, line_nb, element_idx);
-}
-
-
 int obi_column_set_obiint_with_elt_name(OBIDMS_column_p column, index_t line_nb, const char* element_name, obiint_t value)
 {
-	index_t element_idx;
-	element_idx = obi_column_get_element_index_from_name(column, element_name);
+	index_t element_idx = obi_column_get_element_index_from_name(column, element_name);
 	if (element_idx == OBIIdx_NA)
 		return -1;
-	obi_column_set_obiint_with_elt_idx(column, line_nb, element_idx, value);
-	return 0;
-}
 
-
-int obi_column_set_obiint_with_elt_name_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, const char* element_name, obiint_t value)
-{
-	index_t element_idx;
-	element_idx = obi_column_get_element_index_from_name(column, element_name);
-	if (element_idx == OBIIdx_NA)
-		return -1;
-	obi_column_set_obiint_with_elt_idx_in_view(view, column, line_nb, element_idx, value);
-	return 0;
+	return obi_column_set_obiint_with_elt_idx(column, line_nb, element_idx, value);
 }
 
 
 obiint_t obi_column_get_obiint_with_elt_name(OBIDMS_column_p column, index_t line_nb, const char* element_name)
 {
-	index_t element_idx;
-
-	element_idx = obi_column_get_element_index_from_name(column, element_name);
+	index_t element_idx = obi_column_get_element_index_from_name(column, element_name);
 	if (element_idx == OBIIdx_NA)
 		return OBIInt_NA;
+
 	return obi_column_get_obiint_with_elt_idx(column, line_nb, element_idx);
 }
 
-
-obiint_t obi_column_get_obiint_with_elt_name_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, const char* element_name)
-{
-	index_t element_idx;
-
-	element_idx = obi_column_get_element_index_from_name(column, element_name);
-	if (element_idx == OBIIdx_NA)
-		return OBIInt_NA;
-	return obi_column_get_obiint_with_elt_idx_in_view(view, column, line_nb, element_idx);
-}
-

src/obidmscolumn_int.h

@@ -19,7 +19,6 @@
 
 #include "obidmscolumn.h"
 #include "obitypes.h"
-#include "obiview.h"
 
 
 /**
@@ -96,79 +95,5 @@ int obi_column_set_obiint_with_elt_name(OBIDMS_column_p column, index_t line_nb,
 obiint_t obi_column_get_obiint_with_elt_name(OBIDMS_column_p column, index_t line_nb, const char* element_name);
 
 
-/**
- * @brief Sets a value in an OBIDMS column containing data with the type OBI_INT, using the index of the element in the line.
- *
- * @warning Pointers returned by obi_open_column() don't allow writing.
- *
- * @param column A pointer as returned by obi_create_column() or obi_clone_column().
- * @param line_nb The number of the line where the value should be set.
- * @param element_idx The index of the element that should be set in the line.
- * @param value The value that should be set.
- *
- * @returns An integer value indicating the success of the operation.
- * @retval 0 on success.
- * @retval -1 if an error occurred.
- *
- * @since February 2016
- * @author Celine Mercier (celine.mercier@metabarcoding.org)
- */
-int obi_column_set_obiint_with_elt_idx_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, index_t element_idx, obiint_t value);
-
-
-/**
- * @brief Recovers a value in an OBIDMS column containing data with the type OBI_INT.
- *
- * @param column A pointer as returned by obi_create_column().
- * @param line_nb The number of the line where the value should be recovered.
- * @param element_idx The index of the element that should be recovered in the line.
- *
- * @returns The recovered value.
- * @retval OBIBool_NA the NA value of the type if an error occurred and obi_errno is set.
- *
- * @since February 2016
- * @author Celine Mercier (celine.mercier@metabarcoding.org)
- */
-obiint_t obi_column_get_obiint_with_elt_idx_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, index_t element_idx);
-
-
-/**
- * @brief Sets a value in an OBIDMS column containing data with the type OBI_INT,
- *        using the name of the element in the line.
- *
- * @warning Pointers returned by obi_open_column() don't allow writing.
- *
- * @param column A pointer as returned by obi_create_column() or obi_clone_column().
- * @param line_nb The number of the line where the value should be set.
- * @param element_name The name of the element that should be set in the line.
- * @param value The value that should be set.
- *
- * @returns An integer value indicating the success of the operation.
- * @retval 0 on success.
- * @retval -1 if an error occurred.
- *
- * @since February 2016
- * @author Celine Mercier (celine.mercier@metabarcoding.org)
- */
-int obi_column_set_obiint_with_elt_name_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, const char* element_name, obiint_t value);
-
-
-/**
- * @brief Recovers a value in an OBIDMS column containing data with the type OBI_INT,
- *        using the name of the element in the line.
- *
- * @param column A pointer as returned by obi_create_column() or obi_clone_column().
- * @param line_nb The number of the line where the value should be recovered.
- * @param element_name The name of the element that should be recovered in the line.
- *
- * @returns The recovered value.
- * @retval OBIBool_NA the NA value of the type if an error occurred and obi_errno is set.
- *
- * @since February 2016
- * @author Celine Mercier (celine.mercier@metabarcoding.org)
- */
-obiint_t obi_column_get_obiint_with_elt_name_in_view(Obiview_p view, OBIDMS_column_p column, index_t line_nb, const char* element_name);
-
-
 #endif /* OBIDMSCOLUMN_INT_H_ */