161 lines
4.8 KiB
ReStructuredText
161 lines
4.8 KiB
ReStructuredText
*********************************************
|
|
The OBItools3 Data Management System (OBIDMS)
|
|
*********************************************
|
|
|
|
A complete DNA metabarcoding experiment relies on several kinds of data.
|
|
|
|
- The sequence data resulting from the sequencing of the PCR products,
|
|
- The description of the samples including all their metadata,
|
|
- One or several reference databases used for the taxonomic annotation,
|
|
- One or several taxonomy databases.
|
|
|
|
Up to now, each of these categories of data were stored in separate
|
|
files, and nothing made it mandatory to keep them together.
|
|
|
|
|
|
The `Data Management System` (DMS) of OBITools3 can be regarded as a basic
|
|
database system.
|
|
|
|
|
|
OBIDMS UML
|
|
==========
|
|
|
|
.. image:: ./UML/OBIDMS_UML.png
|
|
|
|
:download:`html version of the OBIDMS UML file <UML/ObiDMS_UML.class.violet.html>`
|
|
|
|
|
|
An OBIDMS directory contains :
|
|
* one `OBIDMS history file <#obidms-history-files>`_
|
|
* Two different kinds of directories :
|
|
* OBIDMS column directories
|
|
* OBIDMS column group directories containing OBIDMS column directories
|
|
|
|
|
|
OBIDMS column directories
|
|
=========================
|
|
|
|
OBIDMS column directories contain :
|
|
* all the different versions of one OBIDMS column, under the form of different files (`OBIDMS column files <#obidms-column-files>`_)
|
|
* one `OBIDMS release file <#obidms-release-files>`_
|
|
|
|
The directory name is the column attribute, or sub-attribute if the column directory is in a column group directory.
|
|
|
|
|
|
OBIDMS column group directories
|
|
===============================
|
|
|
|
OBIDMS column group directories contain OBIDMS column directories. They are used to store dictionary-like data, where
|
|
each key corresponds to an OBIDMS column.
|
|
|
|
The directory name is the dictionary attribute. Each key is considered a sub-attribute and is associated to its column.
|
|
|
|
|
|
OBIDMS column files
|
|
===================
|
|
|
|
Each OBIDMS column file contains :
|
|
* a header of a size equal to a multiple of PAGESIZE (PAGESIZE being equal to 4096 bytes
|
|
on most systems) containing metadata
|
|
* one column of data with the same `OBIType <types.html#obitypes>`_
|
|
|
|
|
|
Header
|
|
------
|
|
|
|
The header of an OBIDMS column contains :
|
|
|
|
* Endian byte order
|
|
* Header size (PAGESIZE multiple)
|
|
* Number of lines of data
|
|
* Number of lines of data used
|
|
* `OBIType <types.html#obitypes>`_ (type of the data)
|
|
* Date of creation of the file
|
|
* Version of the OBIDMS column
|
|
* The column name
|
|
* Eventual comments
|
|
|
|
|
|
Data
|
|
----
|
|
|
|
A column of data with the same `OBIType <types.html#obitypes>`_.
|
|
|
|
|
|
Mandatory columns
|
|
-----------------
|
|
|
|
Some columns must exist in an OBIDMS directory :
|
|
* sequence identifiers column (type ``OBIStr_t``)
|
|
|
|
|
|
File name
|
|
---------
|
|
|
|
Each file is named with the attribute associated to the data it contains, and the number of
|
|
its version, separated by an ``@``, and with the extension ``.odc``.
|
|
|
|
Example : ``count@3.odc``
|
|
|
|
|
|
Modifications
|
|
-------------
|
|
|
|
An OBIDMS column file can only be modified by the process that created it, and while its status is set to Open.
|
|
|
|
When a process wants to modify an OBIDMS column file that is closed, it must first clone it. Cloning creates a new version of the
|
|
file that belongs to the process, i.e., only that process can modify that file, as long as its status is set to Open. Once the process
|
|
has finished writing the new version of the column file, it sets the column file's status to Closed, and the file can never be modified
|
|
again.
|
|
|
|
That means that one column is stored in one file (if there is only one version)
|
|
or more (if there are several versions), and that there is one file per version.
|
|
|
|
All the versions of one column are stored in one directory.
|
|
|
|
|
|
Versioning
|
|
----------
|
|
|
|
The first version of a column file is numbered 0, and each new version increments that
|
|
number by 1.
|
|
|
|
The number of the latest version of an OBIDMS column is stored in the `OBIDMS release file <formats.html#obidms-release-files>`_ of its directory.
|
|
|
|
|
|
OBIDMS release files
|
|
====================
|
|
|
|
Each OBIDMS column is associated with an OBIDMS release file in its dorectory, that contains the number of the latest
|
|
version of the column.
|
|
|
|
File name
|
|
---------
|
|
|
|
OBIDMS release files are named with the attribute associated to the data contained in the column, and
|
|
have the extension ``.odr``.
|
|
|
|
Example : ``count.odr``
|
|
|
|
|
|
OBIDMS views
|
|
============
|
|
|
|
An OBIDMS view consists of a list of OBIDMS columns and lines. A view includes one version
|
|
of each mandatory column. Only one version of each column is included. All the columns of
|
|
one view contain the same number of lines in the same order.
|
|
|
|
|
|
OBIDMS history file
|
|
===================
|
|
|
|
An OBIDMS history file consists of an ordered list of views and commands, those commands leading
|
|
from one view to the next one.
|
|
|
|
This history can be represented in the form of a ?? showing all the
|
|
operations ever done in the OBIDMS directory and the views in between them :
|
|
|
|
.. image:: ./images/history.png
|
|
:width: 150 px
|
|
:align: center
|