obitools3/doc/sphinx/source/DMS.rst

*********************************************
The OBItools3 Data Management System (OBIDMS)
*********************************************

A complete DNA metabarcoding experiment relies on several kinds of data.

	- The sequence data resulting from the sequencing of the PCR products,
	- The description of the samples including all their metadata,
	- One or several reference databases used for the taxonomic annotation,
	- One or several taxonomy databases.

Up to now, each of these categories of data were stored in separate
files, and nothing made it mandatory to keep them together.


The `Data Management System` (DMS) of OBITools3 can be regarded as a basic 
database system.


OBIDMS UML
==========

.. image:: ./UML/OBIDMS_UML.png

:download:`html version of the OBIDMS UML file <UML/ObiDMS_UML.class.violet.html>`


An OBIDMS directory contains :
  * one `OBIDMS history file <#obidms-history-files>`_
  * Two different kinds of directories :
     * OBIDMS column directories
     * OBIDMS column group directories containing OBIDMS column directories


OBIDMS column directories
=========================

OBIDMS column directories contain :
    * all the different versions of one OBIDMS column, under the form of different files (`OBIDMS column files <#obidms-column-files>`_)
    * one `OBIDMS version file <#obidms-version-files>`_

The directory name is the column attribute, or sub-attribute if the column directory is in a column group directory.


OBIDMS column group directories
===============================

OBIDMS column group directories contain OBIDMS column directories. They are used to store dictionary-like data, where 
each key corresponds to an OBIDMS column.

The directory name is the dictionary attribute. Each key is considered a sub-attribute and is associated to its column.


OBIDMS column files
===================

Each OBIDMS column file contains :
	* a header of a size equal to a multiple of PAGESIZE (PAGESIZE being equal to 4096 bytes
	  on most systems) containing metadata
	* one column of data with the same `OBIType <types.html#obitypes>`_


Header
------

The header of an OBIDMS column contains :

	* Endian byte order
	* Header size (PAGESIZE multiple)
	* Number of lines of data
	* Number of lines of data used
	* `OBIType <types.html#obitypes>`_ (type of the data)
	* Date of creation of the file
	* Version of the OBIDMS column
	* The column name
	* Eventual comments


Data
----

A column of data with the same `OBIType <types.html#obitypes>`_.


Mandatory columns
-----------------

Some columns must exist in an OBIDMS directory :
    * sequence identifiers column (type ``OBIStr_t``)


File name
---------

Each file is named with the attribute associated to the data it contains, and the number of
its version, separated by an ``@``, and with the extension ``.odc``.

Example : ``count@3.odc``


Modifications
-------------

An OBIDMS column file can only be modified by the process that created it, and while its status is set to Open. 

When a process wants to modify an OBIDMS column file that is closed, it must first clone it. Cloning creates a new version of the
file that belongs to the process, i.e., only that process can modify that file, as long as its status is set to Open. Once the process
has finished writing the new version of the column file, it sets the column file's status to Closed, and the file can never be modified
again.

That means that one column is stored in one file (if there is only one version)
or more (if there are several versions), and that there is one file per version.

All the versions of one column are stored in one directory.


Versioning
----------

The first version of a column file is numbered 0, and each new version increments that
number by 1.

The number of the latest version of an OBIDMS column is stored in the `OBIDMS version file <#obidms-version-files>`_ of its directory.


OBIDMS version files
====================

Each OBIDMS column is associated with an OBIDMS version file in its directory, that contains the number of the latest
version of the column.

File name
---------

OBIDMS version files are named with the attribute associated to the data contained in the column, and
have the extension ``.odv``.

Example : ``count.odv``


OBIDMS views
============

An OBIDMS view consists of a list of OBIDMS columns and lines. A view includes one version
of each mandatory column. Only one version of each column is included. All the columns of
one view contain the same number of lines in the same order.


OBIDMS history file
===================

An OBIDMS history file consists of an ordered list of views and commands, those commands leading
from one view to the next one.

This history can be represented in the form of a ?? showing all the
operations ever done in the OBIDMS directory and the views in between them :

.. image:: ./images/history.png
    :width: 150 px
    :align: center