Compare commits

...

783 Commits

Author SHA1 Message Date
7c2787b6b3 trying to fix cython difficulties 2019-03-26 16:19:55 +01:00
14eca43eac Import taxo 2019-03-26 16:17:44 +01:00
0b4ea49539 Convert relative import an delete cfiles 2019-03-26 16:14:03 +01:00
cd88c37a7e Merge branch 'pip-standard' of git@git.metabarcoding.org:obitools/obitools3.git into pip-standard 2019-03-26 15:54:49 +01:00
1095a617a3 Patch relative import to absolute 2019-03-26 15:54:33 +01:00
5a05258fcb fixed relative cython imports to be absolute 2019-03-26 15:52:59 +01:00
10ab557259 First version of the simplified setup.py script 2019-03-26 15:40:31 +01:00
06178d9d61 Genbank file parser functions that should have been included in a
previous commit
2019-03-20 11:44:43 +01:00
3abe1b7ace obi_errno_to_exception function now properly reads obi_errno global
variable directly
2019-03-20 11:43:12 +01:00
802a3f5933 data import: entries now counted if there are multiple files 2019-03-18 18:16:39 +01:00
7e20870719 Added genbank parser 2019-03-15 16:06:27 +01:00
e8090a44c9 Fixed the ultimate bug with embl (and genbank) parsers: raising any
exception in a python generator makes it unable to resume. So now,
exceptions are not raised but printed, then functions return None and
that's handled at higher level.
2019-03-15 16:06:06 +01:00
832f582802 Fixed no-skip-on-error option :p 2019-03-15 16:04:04 +01:00
58d0c850c2 Made skip on error option True by default...... 2019-03-15 15:50:40 +01:00
7737211ac2 Small fix in embl and genbank features parser 2019-03-15 15:50:11 +01:00
c953f0cb00 Fixed embl import where sequences were not imported as Nuc_Seq objects 2019-03-15 11:41:07 +01:00
bb045c3ae9 added TAXID_COLUMN to C API declarations for Cython 2019-03-15 11:40:06 +01:00
2a4f1b8feb obi import: now properly uses macros for column names 2019-03-15 11:39:21 +01:00
24a63f8732 URIs: URIs built with autocomplete now work too 2019-03-15 10:52:27 +01:00
478d19ab43 Cleaner stderr prints 2019-03-13 18:36:31 +01:00
e3c565d6be Cleaner progress bar 2019-03-13 18:36:05 +01:00
d88390c6d8 Cython API: when importing a file in a DMS, its length is computed
beforehand for the progress bar
2019-03-13 18:35:32 +01:00
50e7cd61a6 added math.h import where needed 2019-03-13 11:17:25 +01:00
49d5f6fb1e removed deprecated comment 2019-03-13 11:17:04 +01:00
b45c2ee653 Cython API: cleaner column rewriting API 2019-03-13 11:13:55 +01:00
6afd1294a7 Cython API: Views: fixed a bug when rewriting a column with different
attributes (last line is not written anymore)
2019-03-12 16:40:30 +01:00
a9ba7744cf obidistutils: added fPIC flag needed for linux compilation and set
minimum python version to 3.7
2019-03-12 14:20:59 +01:00
185a95e667 cleaner Makefile 2019-03-11 15:20:10 +01:00
8835a1a983 removed -R compilaiton flag that gcc doesn't like 2019-03-07 15:55:46 +01:00
1ee50b7222 Fixed a bug when creating a column and checking the comments string if
it was NULL
2019-03-07 15:09:59 +01:00
720bb65b24 Installation: basic Makefile that creates the shared obi3 library used
by Cython
2019-03-07 14:01:37 +01:00
2a1ab9db29 Cython API, Views: guessing an obitype from a python value is now done
through the corresponding functions in utils
2019-03-07 13:57:37 +01:00
4bc52c08c2 minor changes 2019-03-07 13:53:37 +01:00
306da846e3 obidistutils: link obi3 C shared library instead of compiling all C
files with all modules (creating issues with global/static variables).
EXCEPT RUNTIME LINKING DOESN'T WORK YET
2019-03-07 13:50:29 +01:00
af57e532da obidistutils: create doc/sphinx dir if needed 2019-03-07 13:47:23 +01:00
52de6f2717 Update distutils for openmp and new version of pip 2019-02-19 17:30:53 +01:00
29c56572cf Add cfiles everywhere ;-) 2019-02-19 15:04:30 +01:00
de3d12af17 Renamed CAPI file 2019-02-19 14:50:30 +01:00
9ccddd5280 better cfiles 2019-02-19 14:11:29 +01:00
e026e9ec83 Fixed the new alignpaired end to work after ngsfilter with the 9879847
possible cases
2019-02-17 18:32:35 +01:00
4ddd1a1c37 embl iterator: only option on embl directories now works as intended 2019-02-12 16:46:08 +01:00
3015310535 Fixed a bug in kmer similarity computation where the fact that sequences
could be switched was not accounted for
2019-02-10 21:02:24 +01:00
08bcbcd357 ngsfilter: reworked to use apat library 2019-02-06 18:13:54 +01:00
04a3682307 Cython API: added API to use apat (pattern search) C library 2019-02-06 18:12:49 +01:00
6ca6d27ecb ecoPCR: fixed amplicon length computation bug 2019-02-06 18:11:20 +01:00
8f18907566 Cython API: changed revcomp attribute of Nuc_Seq class to is_revcomp to
be more explicit
2019-02-06 18:09:11 +01:00
0b62619e4e Various commentaries and insignificant fixes 2019-01-21 17:32:44 +01:00
c7f5b8d980 Alignpairedend: added alignment using shifting with best kmer similarity
(low level layer in C and Cython API)
2019-01-21 17:30:46 +01:00
59017c0d6b C: taxonomy: fixed a bug when checking for root node 2019-01-21 17:23:25 +01:00
9f6bba183f C: Added a function to get a nucleotide at a specific index in an
encoded sequence
2019-01-21 17:18:02 +01:00
2a6a112d29 obi import: fixed writing quality in views when appropriate (but still
not a satisfying solution)
2018-12-11 19:33:55 +01:00
c437931a35 Cython: fixed history dot graph for all views, and fixed history
recording for build_ref_db and ecotag
2018-12-10 17:09:00 +01:00
eb586b2f53 New command and C functions: obi ecotag 2018-12-09 19:19:32 +01:00
9556130b11 C obi_lcs: updated deprecated column names and associated comments 2018-12-09 19:17:13 +01:00
005aaeec06 C obi_lcs: fixed checking for identical sequences when aligning 2
columns
2018-12-09 19:16:44 +01:00
579f56bb54 obi align (pouic): fixed bug with the saved config when aligning 2
different views
2018-12-09 19:15:58 +01:00
da445066f3 C alignment filter: added a check for sequences not being equal when the
threshold requires that they are
2018-12-09 19:14:51 +01:00
0a407436da C Views: made an error message more specific 2018-12-09 19:14:05 +01:00
54efff36c4 C build ref db: fixed 2 bugs when setting arrays: size of an element in
in bits not bytes and using view API instead of column API
2018-12-09 19:13:06 +01:00
6acb21712a Missing commit for build_ref_db: C API file for cython 2018-12-09 19:11:59 +01:00
12087a6c3a C, views: made 'view_exists' function public (now 'obi_view_exists') 2018-11-27 16:20:30 +01:00
fbabbceb5a Fixed a bug in the array indexer where the value's length was no
properly set to 0 if the value was NA (ignore previous commit with the
same message)
2018-11-27 16:18:34 +01:00
6f27734d71 Cython: fxed a bug in INT tuple columns where values were converted to
double instead of int
2018-11-27 16:14:56 +01:00
b3bfa9ca65 Fixed a bug in the array indexer where the value's length was not
properly set to 0 if the value was NA
2018-11-27 16:12:41 +01:00
ece942e771 new command: build_ref_db to build a reference database with metadata
for the taxonomic assignment of sequences
2018-11-27 16:11:18 +01:00
ef8dc85f3c C, taxonomy: new function to get the lowest common ancestor of two taxa 2018-11-27 16:00:29 +01:00
f942dd856f C: new function to build a reference database with LCA and score
metadata for the taxonomic assignment of sequences
2018-11-27 15:56:50 +01:00
730ea99f85 minor fixes and comments 2018-11-19 11:23:54 +01:00
4d51f4f015 obi import: better checking of whether to import quality 2018-11-19 11:23:39 +01:00
e9c1d5e48d AVLs: made maximum number of nodes per AVL 5 millions, as this combined
with keeping all AVLs mapped seems the most efficient. Now 1 million
sequences more or less constantly takes 1 minute.
2018-11-19 11:22:26 +01:00
7fc1b578cf AVLs: AVLs in a group are not unmapped and remapped constantly anymore
when adding new values, fixed a bug when calculating if an AVL data file
has reached the maximum size, fixed a casting bug, and added a boolean
so read-only AVLs files are not truncated
2018-11-19 11:19:07 +01:00
31053591b5 Fixed 2 bugs when checking qualities matching sequences predicate: now
closing and reopening indexers so that they are mapped properly, and
fixed memory leak when reading sequences
2018-11-19 11:05:53 +01:00
b0da36cb48 New command: obi align, except it's called obi pouic for now because of
a Cython compilation bug
2018-11-07 16:05:48 +01:00
d1f1fd432e Minor fixes 2018-11-07 16:04:17 +01:00
75a28929a7 Renamed Cython alignment library in an attempt to limit some compilation
bugs potentially involving name conflicts
2018-11-07 16:03:32 +01:00
d076ea9900 Alignment: updated functions to align columns (LCS) 2018-11-07 16:00:58 +01:00
6b1c41f3fb Changed an error message to be more specific 2018-11-07 13:37:25 +01:00
362df50fe9 Removed a deprecated element from the DMS structure 2018-11-07 13:36:08 +01:00
b1090574da View import: associated column informations are now correctly updated
with the new versions
2018-11-07 13:35:11 +01:00
8faabd3ebf Cython, URI: Fixed a bug when using an output URI with just a view name
to use the default DMS
2018-11-02 19:04:27 +01:00
35f3e7c30b All commands now handle outputing to another DMS + small fixes 2018-11-02 19:03:09 +01:00
8a8e9e50b2 Fixed declaration going with previous commit 2018-10-31 18:01:04 +01:00
c7ff53b948 obi clean: temporary views are now deleted 2018-10-31 17:52:51 +01:00
1b7bccb236 Small improvement when checking if a view exists 2018-10-31 17:51:10 +01:00
d09aa43133 Cython API: added a function to get the full path to the DMS directory 2018-10-31 14:46:25 +01:00
123e5dc0ac Cython URI API: added an argument to only open the DMS and return the
rest as a character string
2018-10-31 14:45:17 +01:00
320561a582 Views: Added argument to not automatically create default columns in
typed views, a function to delete a view and fixed view history bug
2018-10-31 14:38:05 +01:00
92c0fbc9bf Fixed a bug where an imported column was not flagged as finished,
resulting in its deletion when reopening the DMS.
2018-10-29 17:39:30 +01:00
b11d52d630 Fixed a bug with the DMS counter being wrongly initialized to 0 instead
of 1 (generating memory bugs when using the counter)
2018-10-29 16:12:37 +01:00
6305282305 obi clean: made more efficient with arrays (speed ~x15 compared with
OBI1)
2018-10-21 17:59:02 +02:00
d53323e7f4 Fixed comments bug with obi head and obi tail 2018-10-21 17:39:17 +02:00
e18b762d81 Weird buggy Eclipse commit with nothing changed 2018-10-21 17:35:18 +02:00
0a0f0682a9 Better handling of errors and exceptions when new view name already
exists
2018-10-17 19:47:40 +02:00
4802e32f72 Cython: Sequence objects: repr() method now returns a Fasta or Fastq
formatted string
2018-10-17 16:53:42 +02:00
b027762059 Cython: export: fixed exception raising when no quality data when
exporting to fastq
2018-10-17 16:52:51 +02:00
da0e3d4043 Cython: added full handling of NA strings when importing files 2018-10-17 16:41:15 +02:00
da76f911db Cython: Views: improved repr() method 2018-10-17 15:54:03 +02:00
61ad2deeca obi uniq: Added line breaks when printing informations to cut progress
bar properly
2018-10-17 15:53:28 +02:00
eb6d5581bd Cython: Progress bar: added a cut option to choose whether to do line
breaks every tenth of the full bar, set to False by default for lighter
printing
2018-10-17 15:52:26 +02:00
343dbc7e4d Cython: made the logger lighter (now prints just module name instead of
full module path)
2018-10-17 15:49:55 +02:00
6d018a2d28 Cython: Added 'modulename' in the config informations 2018-10-17 15:47:44 +02:00
2c2df4e098 C: Added a trick to suppress compilation warnings about an unused
function actually called in a macro
2018-10-17 13:13:23 +02:00
8ce6dd6d1a Updated prototypes with no arguments with a void argument as suggested
by compilation warnings
2018-10-17 12:00:40 +02:00
df70086384 New command: obi export 2018-10-17 11:27:50 +02:00
32d8396ee2 Cython: Added fasta and fastq writers 2018-10-17 11:27:15 +02:00
6a8670d24a Cython: minor fixes 2018-10-17 11:26:13 +02:00
ec73fa840a Cython: obi stats fixed to work with reworked options (forgotten in a
previous commit)
2018-10-17 11:25:53 +02:00
11032ec90b Cython: Sequence objects: Quality strings are now returned as bytes
instead of str
2018-10-17 11:24:44 +02:00
8a9ba8b0a8 Cython: Added Column line methods to get a Column line as a str or
bytes, and elements (keys, values) with None values are not returned
anymore
2018-10-17 11:23:07 +02:00
135d3b6e67 Cython: updated the URI decoding to handle outputs other than DMS 2018-10-17 11:21:29 +02:00
58589e04be Cython: rearranged input and output format options to have both and
updated commands accordingly
2018-10-17 11:19:48 +02:00
e6bbe13d81 Cython: fasta and fastq parsers now return bytes and take NA string
argument
2018-10-17 11:16:20 +02:00
61b00d6013 Cython: fastq formatter 2018-10-09 16:41:14 +02:00
8029493c10 Cython: fasta and fastq header formatter 2018-10-09 16:41:00 +02:00
aa5ee53478 Cython: fasta writer 2018-10-09 16:40:30 +02:00
e31c8ea57a New command: obi history to print DMS or view history in bash, dot or
ascii formats
2018-10-07 19:11:36 +02:00
9e700ddc21 obi test: updated to test comments 2018-10-07 19:10:46 +02:00
e9a41c5b97 Commands: updated for JSON formatted comments with history 2018-10-07 19:10:34 +02:00
35cf2962cc Cython: DMS: JSON formatted comments and history handling 2018-10-07 19:06:59 +02:00
74be3c39f0 Cython: Views: JSON formatted comments and history handling 2018-10-07 19:06:23 +02:00
c6ee0bade9 Cython: Columns: goes with handling of JSON formatted comments 2018-10-07 19:04:50 +02:00
ffd5bc76bf Cython utils: functions convert to bytes or str and to remove all empty
objects from a complex object
2018-10-07 19:03:38 +02:00
704d9b0474 Cython: Columns: added support for JSON formatted comments 2018-10-07 18:59:43 +02:00
86bb582a17 Views: implemented handling of JSON formatted comments 2018-10-07 18:56:46 +02:00
bc8c394061 Columns: implemented handling of JSON formatted comments 2018-10-07 18:54:51 +02:00
cef458f570 Obierrno: added errno for JSON related errors 2018-10-07 18:53:53 +02:00
2736a92699 DMS: implemented full information file with JSON formatted comments 2018-10-07 18:53:25 +02:00
79f4185757 C library to handle JSON formatted comments using the cJSON library 2018-10-07 18:51:27 +02:00
1b6b6d825a obi grep: added all the missing filtering options 2018-08-14 17:11:41 +02:00
3847850a9d Taxonomy Cython API: added is_ancestor() function 2018-08-14 17:09:40 +02:00
b57e938cc4 New command: obi stats 2018-08-13 15:08:10 +02:00
2dc7fcceac Minor fixes 2018-08-10 10:39:46 +02:00
e096b929dc New command: obi tail 2018-08-10 10:39:26 +02:00
2c634dae7c New command: obi head 2018-08-10 10:29:37 +02:00
7a4cdc0cfe New command: obi sort 2018-08-09 18:10:47 +02:00
e8dc5eb123 Commands: ngsfilter and alignpairedend can now be used in whichever
order
2018-08-08 19:53:26 +02:00
3fcf29a76f More explicit predicate error when checking that sequences and qualities
match
2018-08-08 19:51:05 +02:00
080a97cccf Cython API: more explicit "Can't guess type" exception 2018-08-08 19:50:26 +02:00
9c9aec2556 Cython API: the associated sequence column for a quality column can now
be specified at the Python level
2018-08-08 19:49:56 +02:00
303648bd47 Cython: embl file parser 2018-07-28 17:14:10 +02:00
2ba6d16147 New command: obi ecopcr 2018-07-28 17:13:45 +02:00
275d85dc5d Cython: fixed a bug when reading an uncompressed file in binary mode
where the first 4 characters would not be read
2018-07-28 17:11:51 +02:00
a39f9697be Views: added macro for taxid column name 2018-07-28 17:10:11 +02:00
b98880b7fa Various non-important fixes and comments 2018-07-28 17:07:17 +02:00
895d09b133 obi import: 'taxid' columns are imported as 'TAXID' to fit view
predicates, and fixed taxdump import and DMS closing
2018-07-28 17:03:00 +02:00
c02c15b93f Cython API: URI decoding now returns the character string with the
object path if it could not be opened
2018-07-28 17:00:42 +02:00
3e8c187f0b Cython API: added EMBL parser and files to import are now read in binary
mode
2018-07-28 16:57:01 +02:00
7f6d1597fc Taxonomy: added functions to check if a taxonomy already exists in a
DMS, and added taxdump import from a compressed file
2018-07-28 16:48:11 +02:00
1de308a856 obi clean: option to only keep heads now works, fixed a bug where last
sequence was not properly labelled, and code is cleaned, fixed and error
checked
2018-05-31 15:11:41 +02:00
892ed83a33 Removed deprecated function declarations 2018-05-31 15:08:11 +02:00
6911bf4d70 obi clean: first version 2018-05-18 14:26:54 +02:00
f0c147c252 C API: Added a function to set an entire column to a specified (atomic)
value.
2018-05-17 15:59:16 +02:00
4aef20add8 Fixed a bug where the line selection column of a view would not be
flagged as finished
2018-05-17 15:17:19 +02:00
62614a8538 Cython API: fixed a bug in URI decoding and option handling where the
quality offset would not be read properly
2018-05-17 15:10:52 +02:00
ffebc6acfb Cython API: better handling of default quality offset value 2018-05-17 15:01:25 +02:00
b91b3176b0 obi uniq: fixed a bug where merged values were wrongly reinitialized 2018-05-17 14:58:15 +02:00
31d8ba5085 obi test: minor change 2018-05-17 14:54:45 +02:00
a166a169cf obi ngsfilter: fixed a bug with -u option 2018-05-17 14:53:53 +02:00
8a10072d99 obi annotate: fixed a bug with --with-taxon-at-rank option and minor
improvements
2018-05-17 14:51:18 +02:00
b380368264 Obi count command 2018-04-04 15:51:23 +02:00
1f4e82e6f6 Fixed three bugs in obi uniq 2018-04-04 15:50:10 +02:00
6825fc13ab Cython API: added ngsfilter file parser 2018-03-21 16:41:25 +01:00
49c17ab7b4 Cython API: added tabular file parser 2018-03-21 16:41:09 +01:00
2684535e26 New command: obi annotate 2018-03-21 16:39:31 +01:00
123fb9d7ba Cython API: in taxonomy, added get_taxon_at_rank() function for Taxonomy
class and rank_idx property for Taxon class
2018-03-21 16:38:26 +01:00
4c3478d8f8 Removed the predicate to check for a quality column (because for example
with obi annotate, clone view so clone predicate, then modify seq, so
quality is deleted, and predicate becomes a problem)
2018-03-21 16:37:19 +01:00
4a815785c4 obi import: added basic taxdump import 2018-03-21 16:35:44 +01:00
75b54c83ca obi grep: fixed bug when reading URIs 2018-03-21 16:34:57 +01:00
53cb3354b8 obi ls command 2018-03-19 13:08:41 +01:00
ea58e254da Cython API: repr function for DMS 2018-03-19 13:08:06 +01:00
9fb63d4894 Minor fixes 2018-03-16 19:05:09 +01:00
d4f7e02c85 New obi grep working with URI API 2018-03-16 19:04:54 +01:00
15e43bb9a1 Cython API: obi import can now import ngsfilter files and tabular files 2018-03-12 18:10:43 +01:00
8a0b95c1d6 New command: obi ngsfilter 2018-03-12 18:09:22 +01:00
dd225a255f obi uniq: better error checking 2018-03-12 18:04:53 +01:00
dad21823ff Cython API: trying to guess the type of a column when adding a None
value does not generate an exception anymore, and RollbackException can
now rollback several views
2018-03-12 18:03:37 +01:00
96bf2daae8 Cython API: added slices in Seq classes and fixes 2018-03-12 17:51:41 +01:00
e6c49b7941 Cython API: moved an eval function to utils 2018-03-12 17:49:54 +01:00
4960662332 Cython API: tobytes() function now handles None values 2018-03-12 17:25:12 +01:00
b2cfa4b52f Cython Sequence classes: reworked improved etc 2018-02-12 14:54:47 +01:00
94a899de12 Cython View API: added small tools 2018-02-12 14:48:27 +01:00
b48330a5c9 Fixed a little bug when cleaning unfinished views 2018-02-12 14:44:56 +01:00
74d880b817 Fixed default quality offset 2018-02-12 14:43:44 +01:00
00993d4215 Cython API: fixed a bug where the quality format would not be read
properly from the configuration values
2018-02-12 14:42:30 +01:00
370fb9272c obi uniq: better typing 2018-02-12 14:38:07 +01:00
c8097e14e1 obi import: removed old traces 2018-02-12 14:36:56 +01:00
01ef85658c New command: obi alignpairedend 2018-02-12 13:30:06 +01:00
f5a00c9322 Cython alignment library 2018-02-12 13:28:20 +01:00
156fb04e88 Implemented functions to build reverse complement sequences 2018-01-05 16:08:36 +01:00
428c4eb5e6 obi import: fixed creation of quality columns (to discuss) 2017-12-19 11:07:00 +01:00
1a5b499b5c Cython API to add an OBI_QUAL column after creating a view 2017-12-19 11:06:24 +01:00
b7b8ba7e5a Better handling of elements names in Cython 2017-12-13 23:12:14 +01:00
e9e7fac999 New obi uniq: stores columns with too many elements per line as
character strings, and keeps a minimum of things in the memory
2017-12-13 22:49:08 +01:00
1fd3323372 Columns: elements names informations are now kept in a memory arena of
adapted size in the header, and added a boolean in the header indicating
whether the values should be evaluated (typically character strings to
be evaluated in Python)
2017-12-13 22:46:50 +01:00
2df5932b67 Cython column API: fixed a memory leak, optimized the reading of
elements names, added a __len__ method to Column_line, and the API for
columns with character strings to evaluate
2017-12-13 22:27:36 +01:00
b93b982a18 Cython: added an option for input taxdump and and an option for the
maximum number of elements in columns with multiple elements per line
2017-12-13 22:25:15 +01:00
ea73047fc7 Added rewinddir before each readdir so that the directories are always
read properly
2017-11-24 18:04:58 +01:00
0998268955 Fixed two little potential bugs when cleaning unfinished columns and
deleted old trace
2017-11-24 18:03:59 +01:00
31726407a3 Taxonomy: fixed a bug where a pointer was not properly reallocated, and
a bug where the merged list of taxids was not built correctly
2017-11-24 18:01:30 +01:00
d21f4a6f90 Header parser: identifiers ending with ';' are now handled 2017-11-24 17:59:52 +01:00
9e3ac477eb OBIDMS: Opened DMS now have a counter associated so that DMS are not
actually opened several times by the same program, which triggers the
cleaning of unfinished views and columns (to discuss)
2017-11-24 17:58:47 +01:00
ee5d647d0d Taxonomy: fixed a bug un parental tree iterator 2017-11-24 17:55:17 +01:00
38fef5b9d4 obi test: better taxonomy testing 2017-11-24 17:54:10 +01:00
3ba7ce1c91 View rollback: version files and column directories aren't deleted
anymore to prevent indexer bug, and fixed a freeing bug
2017-11-15 17:27:26 +01:00
9a50803c00 Added tuple columns containing immutable indexed data arrays of any type 2017-11-15 13:48:59 +01:00
1684f96b79 Fixed a bug when flagging a read-only column as finished 2017-10-26 19:11:29 +02:00
43f65e7fd0 obi uniq: fixed bug where dictionary indexes were not read properly, and
added view rollback in case of an exception.
2017-10-26 19:00:05 +02:00
dfd51939a0 Views are now rollbacked if an error occurs, and unfinished views and
columns are deleted when an OBIDMS is opened.
2017-10-26 18:58:48 +02:00
1ae634d56b Added atexit command to obi import, obi uniq and obi less 2017-10-16 11:09:55 +02:00
04e065094a All DMS opened by a program are now listed and closed with atexit system 2017-10-16 10:35:07 +02:00
5ddd1d9ae6 obi uniq: added taxonomy handling 2017-10-04 16:13:07 +02:00
9fc6868341 Increased maximum length for elements names 2017-10-04 16:10:53 +02:00
f2ece573ff Removed deprecated command 2017-10-04 16:09:41 +02:00
fb9b219abe Fixed a bug with taxonomy URIs not being read correctly 2017-10-04 16:00:30 +02:00
09a5f89849 Column API: improvements to be more flexible when referring to elements
in columns with several elements per line.
2017-10-04 15:59:23 +02:00
535692b020 Taxonomy: new functions and improvements 2017-10-04 15:55:13 +02:00
0ab081f79e Updated obi test to work with changes in taxonomy API 2017-10-04 15:50:32 +02:00
1cb05de7e3 Basic obi less 2017-10-04 15:46:26 +02:00
532d8e9cd7 obi import: small efficiency improvement when dealing with NA values 2017-10-04 15:44:48 +02:00
b4088a7928 Cython API: Added basic taxonomy option 2017-10-04 15:42:17 +02:00
ae24a807da obi uniq: added the option to merge ids, except it only works on small
sets until lists are implemented properly using obiblobs
2017-09-25 17:28:03 +02:00
75c15594c4 obi uniq: added option to use categories additionally to the sequence to
determine uniqueness
2017-09-25 10:56:43 +02:00
5ed6835e0e Fixed a bug where the new line count when truncating a column would not
be computed correctly when dealing with high numbers (bad automatic type
for intermediate result)
2017-09-25 10:52:19 +02:00
41dec03448 Merge branch 'master' of git@git.metabarcoding.org:obitools/obitools3.git 2017-09-18 16:08:31 +02:00
7c57bd33e5 Added check to prevent views from having the name 'taxonomy' (used for
URIs)
2017-09-15 14:54:55 +02:00
a776e46e6d Add the command name in the log 2017-09-15 14:51:13 +02:00
0e140df0fb Cython API: added some imports in __init__ files 2017-09-14 18:30:04 +02:00
4bb071c048 Merge branch 'master' of
git@git.metabarcoding.org:obitools/obitools3.git

Conflicts:
	python/obitools3/commands/import.pyx
2017-09-05 08:59:45 +02:00
5045d0c2e9 xxx 2017-09-05 08:58:07 +02:00
73bca6288f New obi uniq 2017-08-20 18:04:21 +02:00
6a2759eee6 obi import with new input/ouput API 2017-08-20 17:58:36 +02:00
38029b1f77 Forgot a ; 2017-08-20 17:56:18 +02:00
663a1a1091 Cython API: column elements: added possibility to check if an element
exists from its index, and a dict-like get() method
2017-08-20 17:44:05 +02:00
c6d5436a58 Cython API: fixed a bug where iteration on a NUC_SEQS view would not be
done correctly (bug appeared with optimization modifications done
lately)
2017-08-20 17:41:41 +02:00
47cad285d6 Cython API: fixed 2 little bugs in Seq API 2017-08-20 17:39:30 +02:00
74f15d1a23 Cython API: Various fixes in input handlers (parsers, openers etc).
Mostly working but not bug-free
2017-08-20 17:37:51 +02:00
c559ddf487 BUG FIX: creation of a new column would fail because of a case not
handled when a high number of elements per line would imply less than
one line per memory page
2017-08-20 17:30:23 +02:00
93cff94e7f Fixed some compilation warnings 2017-08-20 17:25:58 +02:00
9744a48a67 BUG FIX: seemingly identical obiblobs would have different hash values
because of the padding added by the compiler. Fixed by using calloc
instead of malloc for obiblob memory allocation.
2017-08-20 17:25:15 +02:00
6afdc9fb5f AVLs: Added an error check 2017-08-20 17:21:06 +02:00
6f202363f4 Fixed a typo in doc 2017-08-20 17:20:13 +02:00
7f1ff49aa2 Cython API to import a column and a view from a DMS to another DMS 2017-08-03 16:34:02 +02:00
4b86aa67a8 New C functions to import a column and a view from a DMS to another DMS 2017-08-03 16:33:12 +02:00
a3e81930c2 Views: finished handling and documenting the conditions for an existing
column to be added to a view
2017-08-03 16:32:22 +02:00
644b55b49f Fixed doc typo 2017-08-03 16:29:25 +02:00
927c684fc2 Utils: new function to copy the content of a file into another file 2017-08-03 16:28:54 +02:00
344566d9e9 AVLs: made some functions public and changed some rights to be able to
import AVLs from a DMS to another
2017-08-03 16:27:43 +02:00
407f61a408 Add the possibility to create temporary objects like a temporary
directory and a temporary DMS
2017-07-28 16:33:19 +02:00
09ddd74652 Merge branch 'master' of git@git.metabarcoding.org:obitools/obitools3.git 2017-07-28 15:57:01 +02:00
7c0d882bc9 Patch a bug when creating a DMS not in the current directory. Use the
basename function to locate the DMS name instead of the loop...
2017-07-28 15:56:21 +02:00
35b0c55a8c Cython API: various improvements and checks 2017-07-28 13:15:13 +02:00
b9c65a871f Patch decoding of URL 2017-07-28 12:41:28 +02:00
84bb93096f Cython API: fixes and improvements in Column API 2017-07-28 10:27:04 +02:00
01c69e7e25 Cython API: fixed a bug when printing a column 2017-07-28 10:01:56 +02:00
adf5cbef97 Added DMS method to create a DMS if it doesn't already exists, otherwise
opens it
2017-07-28 09:55:43 +02:00
da48a9d1af Patch group of option : types must be callable not a string 2017-07-28 09:36:18 +02:00
9482c663c0 minor comments and changes 2017-07-27 19:46:34 +02:00
c5f3fdc295 Increased maximum element names length in columns 2017-07-27 19:44:49 +02:00
89e2f80fd8 Goes with previous commit 2017-07-27 19:43:00 +02:00
7112f44fb7 Bug fixes for input handlers, openers, parsers etc. Compiling but not
tested
2017-07-27 19:42:44 +02:00
b2fc1f4611 obi uniq: first version 2017-07-27 19:40:19 +02:00
75f691d55a Cython API: Seq classes reworked 2017-07-27 19:39:58 +02:00
0655063bb0 Cython API: view_NUC_SEQS changes to go with previous commits 2017-07-27 19:39:26 +02:00
9701b1230c Cython API: OBIWrapper.new method is now OBIWrapper.new_wrapper to avoid
mismatching method definitions with subclasses
2017-07-27 19:38:25 +02:00
f8a4428674 Cython API: DMS test_open method doesn't raise an exception anymore 2017-07-27 19:36:28 +02:00
1a0f18a11a Cython API: added a __setitem__ method to the View class that can detect
if the item is a Line and create the corresponding columns if needed +
minor changes
2017-07-27 19:35:28 +02:00
3d7aa52c90 Cython API: Fixed a bug when setting NA values in Column_multi_elts, and
added some properties
2017-07-27 19:31:15 +02:00
69c50ff922 Cython API: added a Column subclass to allow direct access to indexes
for columns that store indexes referring to other data
2017-07-27 19:29:10 +02:00
c91969126b Cython C API declarations to go with previous commit 2017-07-27 19:26:59 +02:00
15d383fa8b Added possibility to specify the offset for encoding and decoding
sequence quality character strings
2017-07-27 19:24:41 +02:00
99ceed5fff Cython API: renamed OBI_Taxonomy to Taxonomy and OBI_Taxon to Taxon 2017-07-27 19:21:45 +02:00
fa8f826cdc Cleanup the end of the file 2017-07-27 16:07:39 +02:00
dc91174a5e Complete the input option group functions 2017-07-27 16:06:48 +02:00
ec65f00cf2 Complete the fasta iterator to manage new input options 2017-07-27 16:05:30 +02:00
8d9cdb4d03 Complete the fastq iterator to manage new input options 2017-07-27 16:05:17 +02:00
949e5f9baf Make a first full version of the URI decoder 2017-07-27 16:04:31 +02:00
3c6a05be54 Add option to the default config corresponding to the parsing of the
inputs
2017-07-27 16:03:47 +02:00
8781ecab1f Add a factory checking the file format and returning the correct
iterator. First version working only with fasta and fastq nucleic
formats
2017-07-27 16:02:52 +02:00
0f6ae7dfa6 Options stuff... ;-) 2017-07-25 13:07:03 +02:00
28259cd88b Beginning of URI decoder -- !!! NOT YET FULLY IMPLEMENTED !!! 2017-07-25 13:05:58 +02:00
b24be84b0a Add a first group of options 2017-07-25 11:14:30 +02:00
59dd0a8a8c Standardized and improved the API to create new columns, updated the doc 2017-07-18 17:34:32 +02:00
c88df2e12c First version of automatic ID and COUNT columns, to discuss (for now,
columns created when NUC_SEQ views are closed if the columns don't
already exist)
2017-07-17 17:31:09 +02:00
1e57bfacb4 Fixed some C documentation 2017-07-17 16:45:08 +02:00
3e6aecc635 Added a C function to add a COUNT column to a view with all lines set to
1
2017-07-11 16:44:23 +02:00
ced9a268a1 obi import: added an option to specify the NA value in the input file
(default is 'NA', same as in R's read.table function)
2017-07-11 12:10:33 +02:00
df2ad41150 Cython APi: Added a width property to views, corresponding to their
column count
2017-07-11 11:46:32 +02:00
f8895e879d Cython API: Added a function to get a column from its index in the view 2017-07-11 11:36:42 +02:00
b729b8928f obi less: fixed bug when the length of a view would be less than the
default number of lines printed
2017-07-10 17:04:02 +02:00
b6b95f26b6 obi import: Skipping sequences is now done through the iterators so that
sequences are not uselessly parsed
2017-07-10 17:02:30 +02:00
b94ec9557f Cython API: None values aren't inclued anymore in the dictionary
returned when getting a line from a column with multiple elements per
line, and reworked that function to be more optimized
2017-07-07 17:28:53 +02:00
143bddf1d1 Cython API: Added an __iter__ method to the class Column_line (iterating
on the elements names) (previously an iteration would work but with
unexpected results)
2017-07-07 15:41:10 +02:00
a718081ebd Bug with error handling: for now obi_errno needs to be passed to the
function handling errors and exceptions, as it can't read the right
value of the global obi_errno (Cython configuration problem?)
2017-07-07 15:36:11 +02:00
740d021276 obi import: fixed bugs when rewriting a column: a bug with new elements
names ignoring previous elements names found, a bug with the global
obi_errno being reset too late, and a bug with the column dictionary
used by obi import not being updated after rewriting a column
2017-07-07 15:33:43 +02:00
906343187b Fixed bug with view option in obi less and obi check 2017-07-06 16:42:27 +02:00
c3cd57a9e3 Removed deprecated file 2017-07-06 10:57:14 +02:00
f03928c679 Committing minor comments before merging branch with master 2017-07-06 10:56:39 +02:00
717ee46f08 Commented a loose print 2017-07-05 18:02:18 +02:00
313508cc94 Better *Seq* classes but still need work 2017-07-05 17:53:46 +02:00
535fc2af83 Column rewriter and optimized View getter 2017-07-05 17:49:05 +02:00
3bbc2ae469 More optimized Column item getter 2017-07-05 17:37:19 +02:00
5ee0b3989a Cython API: set_line of Column_multi_elts now accept as values argument
any class where values are referenced by keys with an iterator
2017-07-05 17:32:32 +02:00
d10192ab0e C functions to detect IUPAC sequences 2017-07-05 17:26:03 +02:00
101f764cce New obi import with rewriting of columns when column type or line
elements (keys) change
2017-07-05 17:15:23 +02:00
cb5ad2ed2d Added functions to try to open a DMS if it exists 2017-07-05 15:38:22 +02:00
f5e992abbf Added a check on the element when setting a value in a column 2017-07-05 14:49:20 +02:00
1d2996c6c0 Better handling and tracing of Index Errors between C and Cython 2017-07-05 14:45:43 +02:00
f6631f3857 Removed deprecated declarations 2017-07-05 14:42:21 +02:00
3f5fef10b9 obi test: minor changes 2017-07-05 14:37:27 +02:00
20c72af697 Basic obi check command to check DMS and view informations 2017-07-05 13:54:19 +02:00
d252131950 Basic obi less command 2017-07-05 13:44:12 +02:00
ca16ce0bb0 Basic obi grep with new Cython API 2017-07-05 11:58:10 +02:00
ac94b35336 Removed unused import 2017-07-05 11:52:31 +02:00
2d65db4ebc Goes with c2af955b : forgotten files for NUC_SEQS views 2017-04-21 15:15:12 +02:00
4b037ae236 Updated obi test to test NUC_SEQS views and the taxonomy API 2017-04-21 12:09:04 +02:00
c2af955b78 Cython view API: added NUC_SEQS views and sequence classes + changed
cloning API
2017-04-21 12:08:14 +02:00
71b1a43df8 Added functions to clone views with a simpler API 2017-04-21 11:58:15 +02:00
1725b8b80c Reworked taxonomy Cython API to be a subclass of OBIWrapper 2017-04-21 11:54:05 +02:00
ab0d08293e Cython API: removed unnecessary imports 2017-04-21 11:51:05 +02:00
2f0c4b90d7 Fixed a problem where a view would have a wrong line count after adding
a first column to it if there was already a Line selection associated
(happening when cloning), and fixed a bad error check.
2017-04-14 16:25:55 +02:00
537b9847da Minor C doc clarification 2017-04-14 16:23:17 +02:00
b998373be5 Cython API: updated the test command for the new API and deactivated the
other commands for now
2017-04-14 16:21:33 +02:00
6f780148e2 Cython API: added taxonomy API 2017-04-14 16:20:30 +02:00
0e08fc486a Cython API: fixed bug when deleting a column from a view where the
Cython wrapper wasn't closed, and fixed the Line selection
materialization
2017-04-14 16:19:18 +02:00
2bbee64e57 Cython API: fixed problems with Column class 2017-04-14 16:14:41 +02:00
693859eec2 Cython API: fixed conversion bugs when setting and getting values
(especially NA values) in OBI_CHAR, OBI_STR and OBI_SEQ columns
2017-04-14 16:07:23 +02:00
a3fad27190 Cython API: automatic importing of column classes now works 2017-04-06 15:45:02 +02:00
f351540b0b Merge branch 'Eric_new_Python_API' of git@git.metabarcoding.org:obitools/obitools3.git into Eric_new_Python_API 2017-04-06 15:39:52 +02:00
6dccaa0213 Patch the registering function : register_all_column_classes 2017-04-06 15:37:51 +02:00
5de9e0de51 Cython API: now using const char* instead of char* for the type of
values read from OBI_STR columns
2017-04-06 15:15:20 +02:00
ad8de80353 Views: better checks when adding an existing column to a view 2017-04-06 14:44:07 +02:00
8cd3e3604f Cython Column API 2017-04-06 14:42:11 +02:00
255f3c92ae Cython View API 2017-04-06 14:41:58 +02:00
08be4e231d Cython Object API 2017-04-06 14:41:43 +02:00
b5b7995411 new Cython DMS API 2017-04-06 14:41:26 +02:00
0dfb1eb3e6 Cython typed columns 2017-04-06 14:40:44 +02:00
381194194c Cython API: compiling but not working 2017-03-06 16:07:02 +01:00
778acc48cd Added linked lists to handle lists of column pointers in views (not
tested)
2017-03-06 16:06:17 +01:00
3319ede837 Views: Column dictionaries now store and return pointers on column
pointers instead of column pointers.
2017-02-22 13:49:50 +01:00
fc20b83ad1 Merging 2017-02-20 14:56:04 +01:00
431c1c8c6a Merge branch 'Eric_new_Python_API' of
git@git.metabarcoding.org:obitools/obitools3.git into
Eric_new_Python_API

Conflicts:
	python/obitools3/obidms/_obidms.pxd
	python/obitools3/obidms/_obidms.pyx
	python/obitools3/obidms/_obidmscolumn_bool.pyx
	python/obitools3/obidms/_obidmscolumn_str.pyx
	python/obitools3/obidms/_obiseq.pxd
	python/obitools3/obidms/_obiseq.pyx
	python/obitools3/obidms/_obitaxo.pxd
	python/obitools3/obidms/_obitaxo.pyx
	python/obitools3/obidms/_obiview.pxd
	python/obitools3/obidms/_obiview.pyx
	python/obitools3/obidms/_obiview_nuc_seq.pxd
	python/obitools3/obidms/_obiview_nuc_seq.pyx
	python/obitools3/obidms/_obiview_nuc_seq_qual.pxd
	python/obitools3/obidms/_obiview_nuc_seq_qual.pyx
	python/obitools3/obidms/capi/obialign.pxd
	python/obitools3/obidms/capi/obidmscolumn.pxd
	python/obitools3/obidms/capi/obitaxonomy.pxd
	python/obitools3/obidms/capi/obiview.pxd
2017-02-20 14:55:36 +01:00
f23315e26f New Cython API: compile but doesn't work 2017-02-17 15:14:06 +01:00
071a3b61ab Merged master fixed conflict. 2017-02-14 10:58:43 +01:00
e524041013 Views: Files for unfinished views now have the extension
'.obiview_unfinished', renamed to '.obiview' when the view is finished.
2017-02-07 17:16:09 +01:00
a9102620f5 Fixed missing email address 2017-02-07 17:14:10 +01:00
7e9932f488 Fixed a C function declaration 2017-02-07 17:12:56 +01:00
e50da64ea1 The elements names when a column contains several elements per line are
now formatted with '\0' as separator and handled in a more optimized way
2017-01-31 16:48:06 +01:00
651c1d7845 utilities: bsearch and qsort with additional user_data pointer argument 2017-01-31 16:45:47 +01:00
c0bcdce724 Taxonomy: documentation for all the functions, and fixed bugs when
closing the taxonomy (overwriting of .pdx files, missing freeing, and
re-placed a misplaced condition)
2017-01-18 18:22:49 +01:00
c065c1914a Taxonomy: adding, writing and reading preferred names, changed some
function names, and fixed a bug with taxa indices not being properly
initialized
2017-01-16 17:28:20 +01:00
0385a92e02 Taxonomy: Refactored the taxdump reading, and little fixes 2017-01-11 16:36:08 +01:00
cf7f2de016 Modify __init__ and close method to deal with registration process 2017-01-10 14:26:16 +01:00
5122ad52a7 Merge branch 'Eric_new_Python_API' of git@git.metabarcoding.org:obitools/obitools3.git into Eric_new_Python_API 2017-01-10 14:07:50 +01:00
4b02ba73ac Add the OBIObject concept 2017-01-10 14:07:10 +01:00
41ad3deec0 Taxonomy: informations about deleted taxids is now read from
delnodes.dmp file and added to *.adx file
2017-01-09 17:28:49 +01:00
d68374018b Taxonomy: functions to read the *.adx file (containing the deprecated
and current taxids and their corresponding indices in the taxa
structure) and to find the taxa using the merged index.
2017-01-06 15:52:21 +01:00
f396625f98 Taxonomy: function to write *.adx files 2017-01-05 15:37:13 +01:00
897032387f Taxonomy: reading merged.dmp file in taxdump 2017-01-05 14:28:36 +01:00
4a1d3167a7 Last change on my branch 2017-01-02 16:46:52 +01:00
153c22257f Last change on my branch 2017-01-02 16:46:17 +01:00
2139bfc748 refactoring... 2017-01-02 13:05:22 +01:00
65f3b16e6d Refactoring ... 2016-12-29 18:22:05 +01:00
0526386337 first working DMS class 2016-12-27 06:17:45 +01:00
62caf1346e temporary remove some files 2016-12-26 15:03:24 +01:00
3ac6e85fb3 Big refactoring 4 2016-12-26 14:58:03 +01:00
5156f6bb9e Big refactoring 3 2016-12-26 14:18:01 +01:00
e6db2086d5 Big refactoring 2 2016-12-26 13:56:31 +01:00
daacd0df76 Strong refactoring 1 2016-12-26 13:35:31 +01:00
8e92bf6dac LCS alignment: it is now checked that sequences are not longer than what
a 16 bits integer can code for (as the LCS and alignment lengths are
kept in 16 bits registers)
2016-12-22 17:06:23 +01:00
30e4359c85 LCS alignment: documentation for all the lowest level functions 2016-12-22 17:03:51 +01:00
5c50e5b378 Embryo of code for openMP parallelization of LCS alignment but
deactivated for now because can't make it compile with cython/clang
2016-12-20 11:46:58 +01:00
3cedd00d7f Add register function for column type 2016-12-20 11:13:57 +01:00
82fbe43980 transfert method to obiviews 2016-12-20 08:18:47 +01:00
d1a972dfcb patch import 2016-12-20 08:15:42 +01:00
f43dc3e3ab separate the obicolumn classes in new files 2016-12-20 08:15:08 +01:00
9c71b06117 Removed deprecated TODOs 2016-12-19 14:36:40 +01:00
3bf5260174 Merge branch 'master' of git@git.metabarcoding.org:obitools/obitools3.git 2016-12-19 10:31:18 +01:00
857a5198e4 Updated `obi lcs` for the LCS alignment of two columns 2016-12-16 19:40:36 +01:00
d99447c12b C function for LCS alignment of two columns, and optimized and fixed
line count bug in function to align one column
2016-12-16 19:39:02 +01:00
303bd6f445 Added function to build kmer table for 2 columns, and fixed bug (with
line count) when building kmer table of one column
2016-12-16 19:10:18 +01:00
490f5fe6b9 Updated deprecated code in cython API for columns (using line count of
view instead of column)
2016-12-16 19:04:21 +01:00
191c83aafc Added missing *.cfiles 2016-12-15 15:28:34 +01:00
04d39c62ab Try for a new API 2016-12-14 08:44:44 +01:00
9b24818fe2 Refactored alignment code for minimum redundancy between the function
that aligns 1 column and the function that aligns 2 columns
2016-12-13 17:18:12 +01:00
06cb7a9a58 Some change in the way to manage access to special items of the
dictionary like sequence or quality
2016-12-13 12:49:34 +01:00
fc55fc117d Some cosmetic on the code 2016-12-13 12:48:13 +01:00
4ef5cb0d87 Move the OBIView_NUC_SEQS class to files _obiview_nuc_seq.pxd and
_obiview_nuc_seq.pyx to avoid circular inclusion
2016-12-13 12:46:49 +01:00
fc805e5443 Remove some warnings in the editor 2016-12-13 08:29:22 +01:00
8d7ef7d3d1 patch the distutils to add the C source directory in the include path.
This should solve most of the compilation problems related to .h files
located in this directory
2016-12-13 08:02:09 +01:00
8afb1644e9 Alignment: API rework. 'obi align' is now 'obi lcs', and the results are
now written to columns automatically created in the output view, all
optimally handled at the C level.
2016-12-12 11:58:59 +01:00
fa4e4ffaff Changed the cython API to create new views so as to have different
functions for the different cases
2016-12-07 14:17:57 +01:00
936be64c34 Goes with 5e0c9f87 (missing ';' and fixed compilation warnings) 2016-12-05 11:18:29 +01:00
5e0c9f878b Added the doc for the function building the element names, and a missing
free
2016-12-05 10:46:21 +01:00
852e5488c8 The default element names for columns with multiple elements per line
are now "O;1;2;...;n"
2016-12-02 17:54:51 +01:00
e60497651c Updated the documentation for the functions to set and get in the
context of a view
2016-11-30 12:22:47 +01:00
4ad8c16a73 Finished adding all the functions to directly set and get indices in
columns containing indices referring to any type of data.
2016-11-30 11:08:11 +01:00
6f6099687d Sequence alignment: if no sequence column is given and the view has the
type NUC_SEQS_VIEW, the default sequence column is aligned
2016-11-29 16:52:41 +01:00
98d0849653 Sequence alignment: added the possibility to specify the index of the
sequences to align in a column containing multiple sequences per line (C
level for now)
2016-11-29 16:15:02 +01:00
5fb025f310 When aligning, it is now quickly checked whether the sequences are
identical using their indexes
2016-11-28 11:39:29 +01:00
8ce6f6c80b Added an argument to specify whether the two sequences can be identical
when applying filters before aligning
2016-11-28 11:38:02 +01:00
3e53f9418b Added functions to recover the indexes themselves from any column
referring to indexed values
2016-11-28 11:35:19 +01:00
d40d2d0c76 Fixed error in documentation 2016-11-28 10:55:23 +01:00
f897e87600 When closing a view, it is now automatically checked that all OBI_QUAL
columns correspond to their associated OBI_SEQ column
2016-11-25 12:04:57 +01:00
70e056a2aa It is now impossible to open or clone a view that is not finished (= has
been closed at least once)
2016-11-24 11:19:07 +01:00
8abbfa203a Good file for commit 6fa9a8bd: When a view is cloned, a comment is added
to the new view specifying the name of the cloned view
2016-11-23 11:32:39 +01:00
6fa9a8bd76 When a view is cloned, a comment is added to the new view specifying the
name of the cloned view
2016-11-23 11:29:21 +01:00
76a4c6b14e Fixed a bug when cloning a view and checking its type 2016-11-23 11:28:17 +01:00
0ab9e6c05a When adding an existing column to a view, it is checked that the
column's line count is at least the view's line count. This can't be
more stringent for reasons that need to be rediscussed
2016-11-23 11:04:53 +01:00
70c49e214a Added the kmer filter to LCS alignments, and now obiblobs containing
encoded sequences are directly put in int16_t arrays for the alignment
2016-11-18 16:29:28 +01:00
08e67a090f Changed the inline functions syntax, which should make it compatible
with more compilers
2016-11-18 16:21:26 +01:00
621b4972db Functions to get obiblobs through views 2016-11-18 15:59:50 +01:00
7d022c1a52 If the indexer name is NULL when creating a column, it now becomes the
column name
2016-11-18 15:56:51 +01:00
1c71c195fc Goes with a0ebc2d8 2016-11-10 15:01:29 +01:00
54cfeffd85 Goes with 8f724f4f, forgotten file 2016-11-10 14:48:31 +01:00
a0ebc2d871 Functions to directly retrieve Obiblobs from indexers 2016-11-10 14:45:28 +01:00
8f724f4f8e Some code refactoring 2016-11-09 16:48:00 +01:00
359578814b Added view type property to OBIView cython class and updated obi export
to use it
2016-11-08 17:49:59 +01:00
51b23915ca Added properties for Nuc_Seq cython classes (and updated commands using
them)
2016-11-08 16:59:32 +01:00
b5b889c4a2 Fixed the OBI_Nuc_Seq_Stored cython class not being up to date with the
new properties of its parent class
2016-11-08 11:26:37 +01:00
36ac315125 Fixed bugs with python view type when creating a new view, and a bug
when trying to guess the obi type of a nucleotide sequence when its type
was bytes
2016-11-08 11:23:54 +01:00
8291693309 obi grep: updated to work with the new line selection class and within
the local sequence environment, and progress bar functioning
2016-11-08 11:19:12 +01:00
4bc19c3e49 obi export: view type is now checked and progress bar functioning 2016-11-08 11:17:20 +01:00
2d2fe5279d Added functions to add new taxa to a taxonomy with handling of
associated *.ldx files
2016-11-03 17:59:21 +01:00
2504bf0fa9 Added an iterator to the OBI_Taxonomy cython class 2016-11-02 11:08:18 +01:00
d8a257e711 Taxonomy handling functions in C. Features: read taxdump, read binary
files, write binary files. Not fully handled yet: *.adx, *.pdx, *.ldx,
merged.dmp and delnodes.dmp files.
2016-10-27 18:56:11 +02:00
b63d0fb9fb Added C functions to write .rdx, .tdx, .ndx binary taxonomy files from a
taxonomy C structure
2016-10-14 17:03:10 +02:00
0dfd67ec89 The endianness of binary taxonomy files is now correctly checked 2016-10-10 17:04:29 +02:00
0faaac49cf The taxonomy directory of the DMS is now automatically created with the
DMS
2016-10-10 17:02:51 +02:00
1b07109e51 Removed deprecated code 2016-10-10 17:01:51 +02:00
60ab503a14 Added properties in the OBI_Taxonomy class 2016-10-10 17:01:17 +02:00
2dcfdc59fc When a new view is created with a line selection, the view to clone is
automatically found + compacted redundant code + fixed potential bug
when cloning a NUC_SEQS view by name
2016-10-06 17:55:18 +02:00
399fc2c051 Removed deprecated source files previously used for tests 2016-09-30 17:49:37 +02:00
9cd57deca9 Added OBIView_line_selection class to make new line selections
associated with the view to clone, and improved and renamed method
closing a view
2016-09-30 17:48:53 +02:00
d88811ed7d Added a seed option to the obi test command for reproducible tests 2016-09-29 17:34:48 +02:00
8c402101e4 Renamed private attributes as _* and removed some deprecated code 2016-09-28 16:56:44 +02:00
1a7b42018e Added some error checking when opening or creating a view 2016-09-28 14:28:34 +02:00
b717e8bb8b Added properties for the OBIView class and cleaned up deprecated code 2016-09-28 14:26:23 +02:00
03a2c8ef7c Finished restructuring the OBIDMS_column class properties 2016-09-27 14:16:30 +02:00
a7f891d1c9 Added a lines_used property to the OBIDS_column class 2016-09-26 18:04:28 +02:00
bd50b3f972 Added version property to OBIDMS_column class 2016-09-26 17:45:10 +02:00
81380363b7 Added original_name property to OBIDMS_column class 2016-09-26 17:31:32 +02:00
a4b8349274 Added data_type property to OBIDMS_column class 2016-09-26 17:12:20 +02:00
a474391b27 Added nb_elements_per_line property to OBIDMS_column class 2016-09-26 17:01:13 +02:00
a0bc45cc92 Added elements_names property to OBIDMS_column class 2016-09-26 16:53:16 +02:00
76f89717fe Added alias property to OBIDMS_column cython class 2016-09-26 16:12:48 +02:00
b408a4f6eb Changed file name limits to adapt to system limits + minor changes 2016-09-22 18:05:07 +02:00
b083745f56 Deleted the "new line selection while editing a view" system 2016-09-22 11:19:29 +02:00
43f3c69a40 Fixed bug when cloning column with line selection 2016-09-21 17:50:21 +02:00
e79507b629 Fixed bugs in the process ensuring that all the columns of a view have
the same line count, fixed a bug when trying to set a value in a view
when a line selection exists, fixed a bug when adding a new column to a
view where line counts would be wrong
2016-09-21 17:42:17 +02:00
bb25723d99 Improved documentation of a function 2016-09-21 17:30:39 +02:00
a0da984003 Fixed bug where columns would not get truncated to the right size, and
fixed bug where column directories would be open and not closed in some
instances
2016-09-21 17:28:52 +02:00
802bae110b Removed deprecated function 2016-09-21 17:09:59 +02:00
dd55aef3e5 Added column class method to get the unique references (name and
version) of a column
2016-09-21 17:08:44 +02:00
9ac522fde1 Better obi test command 2016-09-21 17:06:35 +02:00
6adb9eb623 Should solde issue #56 2016-09-19 21:40:40 +02:00
8f49553d5a First version of the obi test command, testing that the OBITools3 work
correctly
2016-09-15 12:26:07 +02:00
986f90c59e Fixed bug where column directories weren't closed correctly, leading to
too many file descriptors open, and added error checking when closing
file descriptors
2016-09-15 12:18:40 +02:00
a240ec0169 Added error checking when closing file descriptors 2016-09-15 11:58:56 +02:00
0a3c23d9d0 Added a missing closedir 2016-09-15 11:58:34 +02:00
8724445fa1 Added error checking when closing files 2016-09-15 11:50:30 +02:00
de189fd7e0 Fixed major bug when cloning an AVL where the bloom filter was not
copied properly (because the sutructure copy via assignation does not
work for structures with a variable size)
2016-09-15 11:47:02 +02:00
9a97f1f633 View predicates are now carried over when cloning a view 2016-09-06 16:22:24 +02:00
00014eb023 View files now have the *.obiview extension 2016-09-06 14:19:13 +02:00
acc0da2d0b Readjusted some limits for file names and file numbers to be under OS
limits
2016-09-05 12:39:04 +02:00
668696fc5a Fixed major bug: when setting all the columns of a view to the same
number of lines, columns are now cloned before being enlarged if needed
+ predicate functions now print error messages if the predicates are not
respected
2016-09-05 12:37:36 +02:00
ba84ef4847 Fixed typo 2016-09-05 12:31:06 +02:00
c9dce03295 Fixed major bug when cloning an AVL group (last AVL of new group was not
correctly enlarged before copying the data) + minor improvements
2016-09-05 12:29:52 +02:00
eb82d088cb Added some view class methods 2016-09-05 12:20:00 +02:00
f46ea0b988 Finished fixing issues with DMS paths 2016-08-30 11:09:45 +02:00
5b2e370ffb Fixed a bug when using an absolute path for a DMS 2016-08-29 17:30:31 +02:00
8d360b0fac Minor improvements to obi export command 2016-08-19 17:49:22 +02:00
b34769b27c Minor improvements to obi export command 2016-08-19 17:46:55 +02:00
2d0a714e37 Basic obi export command exporting from view to fasta or fastq format,
for testing purposes
2016-08-19 17:40:58 +02:00
7b780ffb28 View files now have a dynamic size to allow unlimited comments size 2016-08-18 17:57:03 +02:00
e4129610cf Quality columns are now optional in NUC_SEQS views + minor fixes 2016-08-16 15:17:26 +02:00
cf839522e7 Minor update and fix to obi grep command 2016-08-12 17:45:44 +02:00
10b22f79da The cython subclass is now correctly chosen when cloning a view 2016-08-12 17:39:19 +02:00
ad8e10f2d1 Reworked a bit alignment API 2016-08-12 15:56:07 +02:00
92cad61417 Fixed bug when closing views with no associated predicate 2016-08-12 15:52:38 +02:00
64a745ce0b First very basic version of obi grep command 2016-08-11 17:32:08 +02:00
2d8ac2b035 Fixed bug when creating an OBI_IDX column 2016-08-11 17:30:32 +02:00
5b7917bb5a Fixed bug when writing predicates in view file 2016-08-11 17:30:09 +02:00
d3c58780a0 Added __len__ function do OBIViews that returns the line count 2016-08-10 17:20:23 +02:00
029d395da1 Added __iter__ function to OBIView lines 2016-08-10 17:08:22 +02:00
bea02cc7a5 Added (temporary?) check for the type of quality strings because the
import now seems to return them with bytes type
2016-08-10 16:25:45 +02:00
4ba01617af Fixed obscure compilation bug 2016-08-10 15:26:40 +02:00
bec684d5e2 Fixed merge conflict 2016-08-10 15:05:37 +02:00
2aaa87edcc 1st version of obi align command and reworked functions that handle
column alignment
2016-08-10 14:51:02 +02:00
400a3f9f3d Merge branch 'Eric_version_for_sequence'
Conflicts:
	python/obitools3/obidms/_obidmscolumn_seq.pyx
2016-08-04 09:42:42 +02:00
d1d26b9028 Simplify the code 2016-08-04 08:00:54 +02:00
465ea81c77 Merge branch 'master' of git@git.metabarcoding.org:obitools/obitools3.git 2016-08-03 10:13:47 +02:00
1e6d6e32e0 Switch to Cython version >= 0.24 2016-08-03 10:13:10 +02:00
ccc877764e Patch a bug in the printing of the progress bar leading to a bus error
when compiled with some C compilers and Cython >= 0.24
2016-08-03 10:12:23 +02:00
8f0462c407 Merge branch 'master' into Eric_version_for_sequence
Conflicts:
	python/obitools3/obidms/_obidmscolumn_seq.pyx
2016-08-03 10:09:20 +02:00
26b8e1f215 Modified C API to set and get in columns: added functions to set and get
using column names instead of pointers, and changed function names
2016-08-02 16:33:19 +02:00
312f50ff0f Major update: Column aliases. Columns are now identified in the context
of a view by an alias that can be modified.
2016-08-01 18:25:30 +02:00
3843485a04 Deleted deprecated function declaration that would make compilation
impossible and fixed error in documentation
2016-07-22 16:21:02 +02:00
20425a5d2b Deleted deprecated structure declarations 2016-07-19 15:48:56 +02:00
56e4848ebd The predicates associated with a view are now described in its comments
field
2016-07-19 15:31:21 +02:00
8850e40b6e Minor changes for better presentation 2016-07-19 15:30:17 +02:00
b89af38109 Goes with 38718320 2016-07-18 13:57:49 +02:00
38718320f9 First version for the association of one column to another. Closes #55 2016-07-15 15:38:49 +02:00
8ee85c3005 A first version of predicate functions that are checked when a new view
is saved and closed
2016-07-12 14:54:11 +02:00
000b9999ad Merge branch 'master' of git@git.metabarcoding.org:obitools/obitools3.git 2016-07-03 09:22:22 +02:00
aff9831c13 Substitute fprintf call by fputs call to conform with the new ubuntu
compilation rules
2016-07-03 09:21:56 +02:00
448fa8d325 first trial for a fasta formater 2016-07-03 09:18:52 +02:00
6af62d8124 Change a fprintf without argument to a fputs to comply with the new
default parameter on ubuntu
2016-07-03 08:25:06 +02:00
0869b9ba3f Closes issue #47 by storing each view in a separate file named with the
view's name and created upon view creation.
2016-06-30 11:41:30 +02:00
ad2af0b512 Some comments updated 2016-06-16 11:26:54 +02:00
38e603ed57 Deleted some redundant cython code 2016-06-10 10:34:47 +02:00
f438c3d913 OBIQUAL columns can now handle multiple elements per line 2016-06-09 15:54:36 +02:00
2a1ea3ba3f Setting NA values is now handled properly for OBI_SEQ, OBI_STR and
OBI_QUAL columns
2016-06-09 14:22:36 +02:00
fc3641d7ff Read-only AVLs are now hard-linked instead of copied when cloning an AVL
group to make it writable. Also fixed several bugs when handling AVL
groups.
2016-06-03 19:02:46 +02:00
799b942017 Deleted old debugging print 2016-06-03 18:57:32 +02:00
6e3f5b230e Fixed typo in doc 2016-06-03 18:56:45 +02:00
2f57f80c63 Fixed a bug where an unmapped variable would be read 2016-06-03 18:55:58 +02:00
2962c4d250 Goes with previous commit 2016-06-03 18:54:25 +02:00
69bf7ec2e7 NA value for OBI_STR and OBI_SEQ columns is now NULL 2016-06-03 18:53:22 +02:00
bac7ce7184 Start of the implementation of the export methods 2016-06-02 19:10:33 +02:00
f186395661 Trap potential exception generated by char* to bytes casts 2016-05-29 21:18:20 +02:00
85395dfc1a value returned for sequence is now bytes and no more str 2016-05-29 13:53:32 +02:00
f830389974 Add some comment on the location of the align method. 2016-05-29 12:58:31 +02:00
2e35229357 Add conversion checking on the value of a seq column 2016-05-29 12:54:13 +02:00
a8ed57dc6e few small changes 2016-05-21 12:29:55 +02:00
c3274d419c remove an extra debug log 2016-05-21 12:29:08 +02:00
cca0dbb46b Close issue #54 by adding a read1 method to the MagicKeyFile class 2016-05-21 12:24:48 +02:00
5a78157112 increase parsing speed of the header 2016-05-21 10:29:11 +02:00
0b9a41d952 Patch a bug about the reading of the last sequence 2016-05-21 10:28:03 +02:00
e681ca646d Fixed a problem with some columns being shorter in views and triggering
errors when trying to get values. Temporary fix that needs discussion
2016-05-20 18:45:29 +02:00
3b59043ea8 Major update: New column type to store sequence qualities. Closes #41 2016-05-20 16:45:22 +02:00
ffff91e76c Fixed variable name that had been accidentally changed for better
clarity
2016-05-18 13:27:41 +02:00
6a8df069ad Indexers are now cloned if needed to modify them after they've been
closed. Obligatory indexers' names now follow the same pattern as other
indexers (columnname_version). Closes #46 and #49.
2016-05-18 13:23:48 +02:00
8ae7644945 First version of quality handling (not working yet) and now it is
checked that a column is writable before enlarging it
2016-05-11 16:38:14 +02:00
b3c47809da First version of alignment functions (imported from suma* programs) 2016-05-11 16:36:23 +02:00
3567681339 Now when a column is added to a view, if there is a line selection, all
columns in the view are cloned first
2016-05-11 16:34:20 +02:00
757ef8509a Deleting CeCILL license duplicates 2016-05-09 11:17:45 +02:00
f961621f5d Minor improvements in _obidms Cython layer 2016-05-04 13:43:26 +02:00
bc12360490 Reworked and commented a bit the cython layer for dms, columns and views 2016-05-02 15:16:06 +02:00
872071b104 Removed a list of column pointers kept in the OBIView class that was not
really needed
2016-05-02 14:23:42 +02:00
32cc8968e8 Adding CeCILL license 2016-05-02 11:51:59 +02:00
d6481f0db8 Merge branch 'master' of git@git.metabarcoding.org:obitools/obitools3.git 2016-04-29 17:46:59 +02:00
a32920e401 Relative paths when creating or opening a DMS now work 2016-04-29 17:46:36 +02:00
31cf27d676 Added indexer function that returns the name of the indexer 2016-04-29 16:18:56 +02:00
baba2d742e commenting _obidms.pyx 2016-04-29 16:07:03 +02:00
5bd12079ae Added comments about listing columns and indexers in obidms functions 2016-04-29 16:06:01 +02:00
072ee5ac03 Re-re-fixed line breaks in README file 2016-04-29 15:44:40 +02:00
9fe21316ff Refixed line breaks in README file 2016-04-29 15:39:46 +02:00
3dc3aaa46b Fixed line breaks in README file 2016-04-29 15:36:58 +02:00
b371030edd Adding README file 2016-04-29 15:35:08 +02:00
b3976fa461 Merge branch 'luke_tests' 2016-04-28 11:17:24 +02:00
6ea2cfb9ca Merging luke_tests branch without the commit turning inline functions in macros 2016-04-28 11:17:18 +02:00
0eca86107e Pseudo obihead for tests 2016-04-27 14:27:28 +02:00
0de953a3ef pseudo obigrep for tests 2016-04-27 14:19:55 +02:00
f3b20b809d Fixed bug with indexer names being defined and generating seg fault if
creating a column not using indexers
2016-04-27 14:01:36 +02:00
d159b921eb Fixed obi import trying to print all lines at the end (source of
segfault?)
2016-04-27 13:14:19 +02:00
4e4cf46b16 Added all C files as source files for all cython files to stop having
that kind of problem with linux systems
2016-04-27 10:44:24 +02:00
6b61533650 Added more C source files for _obiseq 2016-04-27 10:41:00 +02:00
419885485b Added files in _obitaxo C sources for cython 2016-04-27 10:30:16 +02:00
0c8504b6db Commented #ifdef directive for detect_bucket_size function because it
causes errors
2016-04-27 10:24:40 +02:00
654c34a1a6 changed inline functions to macros to make it work on Luke 2016-04-26 15:40:12 +02:00
2d8c06f7b7 Fixed variable initialization for error detection 2016-04-26 14:38:46 +02:00
a6c8d35491 import command a bit modified for tests 2016-04-26 14:29:54 +02:00
366264828e Renamed MurmurHash2.c file to murmurhash2.c as it could be a problem 2016-04-26 14:29:17 +02:00
d3a6ff6043 Removed deprecated code 2016-04-26 14:27:16 +02:00
5ca84b91dc Merge branch 'master' of git@git.metabarcoding.org:obitools/obitools3.git 2016-04-25 18:35:57 +02:00
87935c6678 Fixed all compilation problems with new function names, locations etc 2016-04-25 18:35:02 +02:00
92980508c0 Made the function to clone a column in the context of a view private 2016-04-25 18:15:25 +02:00
65880db422 Made function to update the line count of a view private 2016-04-25 18:11:37 +02:00
767d9c7804 Reordered view functions for better coherence 2016-04-25 18:07:58 +02:00
2566377e2a Updated the documentation for utils functions 2016-04-25 18:02:58 +02:00
1fbbdd43f9 Updated obiversion_t declaration 2016-04-25 17:58:37 +02:00
8cdfbb379e Documentation for views and reworked the code a little 2016-04-25 17:58:12 +02:00
0a55e26520 Reworked obiview code and added more comments 2016-04-25 11:37:53 +02:00
68a8509c12 Updated documentation in obitypes.h 2016-04-25 10:33:01 +02:00
5f98d2ed5c Fixed the calculation of the size of data for OBI_STR and OBI_SEQ
columns
2016-04-25 10:26:51 +02:00
ef1be141c1 Update Licence to english version 2016-04-23 18:03:50 +02:00
bbfd40d56d Add license 2016-04-23 18:03:10 +02:00
5d08da46a2 Updated the documentation in obidmscolumn.h 2016-04-22 17:55:53 +02:00
66045acf1d Creating a column now uses the function to create the indexer name if
one was not provided
2016-04-22 17:47:00 +02:00
6977c4315c Improved function to build an indexer name 2016-04-22 17:38:23 +02:00
839b3000a8 Added a function to build indexer names 2016-04-22 17:08:23 +02:00
ffa4557928 changed MAP_PRIVATE flags to MAP_SHARED when opening a column because it
seems a lot more efficient
2016-04-22 16:26:24 +02:00
003cd11362 Fixed initialization of NA values for OBI_STR and OBI_SEQ columns 2016-04-22 16:14:23 +02:00
c87227b65a Uncommented an error message that doesn't need to be commented anymore 2016-04-22 16:11:56 +02:00
c07e75f2ac Updated the documentation for OBI_STR columns 2016-04-22 15:59:32 +02:00
6b394a5cf7 Updated the documentation for OBI_SEQ columns 2016-04-22 15:58:20 +02:00
2416b8ccd8 Deleted more unused inclusions in OBI_STR and OBI_SEQ column types code 2016-04-22 15:56:09 +02:00
b9921e111d Removed unused inclusions and definitions in all column types code 2016-04-22 15:50:19 +02:00
8f5aa8841d Removed unused definition in OBI_IDX columns code 2016-04-22 15:44:30 +02:00
900d67de87 Updated the documentation for columns with the type OBI_IDX 2016-04-22 15:43:39 +02:00
22e3c3eeed Updated the documentation for obidms functions 2016-04-22 11:28:09 +02:00
4ead37ee48 Finished moving obiblob functions to obiblob files and documentation for
obiblob functions
2016-04-21 15:18:14 +02:00
bce360bbd5 Documentation for obiblob indexer API 2016-04-21 15:08:40 +02:00
2a68cb26f8 Improved AVL tree documentation 2016-04-21 15:07:27 +02:00
043e70ff49 Updated AVL documentation 2016-04-21 14:39:03 +02:00
66021367f6 Moved some blob functions to obiblob.c 2016-04-21 14:20:26 +02:00
e69f44ae3d Little annotations for the murmur hash function. 2016-04-21 13:53:29 +02:00
1941a3785e Updated encode functions documentation 2016-04-21 13:46:02 +02:00
c7b8db6a2e Replaced malloc+memset with calloc 2016-04-21 13:45:39 +02:00
1dc4a3be49 Documentation for DNA sequence indexing functions 2016-04-21 13:36:51 +02:00
09597016fd Short doc for crc function 2016-04-21 13:23:52 +02:00
1a2fa0923c Documented the functions indexing and retrieving character strings 2016-04-21 11:35:21 +02:00
00f2f2cc51 Documented changes made in bloom functions 2016-04-21 11:22:31 +02:00
7a88ca619a First obi import (doesn't import tags yet because NA values aren't
handled)
2016-04-15 17:00:08 +02:00
eddd19a245 Changes in obi commands 2016-04-15 16:59:21 +02:00
2aafecc3b5 Changed sequence 'description' to 'definition' everywhere 2016-04-15 16:31:43 +02:00
094b2371e9 Deleted obsolete directory 2016-04-15 14:44:31 +02:00
c1034d300d merging and fixed git conflict with obiavl.h 2016-04-15 13:23:29 +02:00
02d67c257f The default name of an AVL is now the column name + '_indexer', and when
an AVL is opened (as opposed to created), it is read-only
2016-04-15 12:55:26 +02:00
e04ea85d1e Fixed problematic __str__ method and useless declarations in the
OBI_Nuc_Seq_Stored class
2016-04-15 11:22:05 +02:00
527d3555f0 Moved the functions getting full paths for files and directories to
obidms.c/.h files
2016-04-15 11:11:13 +02:00
71492ad229 Made the handling of listing and unlisting opened columns and indexers
functions in the obidms files.
2016-04-15 10:49:12 +02:00
73d64e5aff Renamed 'unmap_header' function to 'close_header' 2016-04-14 15:19:27 +02:00
4cb52e1632 Made the truncating of columns automatic when closing them (note:
already the case for AVLs)
2016-04-14 15:13:30 +02:00
9d042f7bd0 Refactored and relocated the set and get functions of all column types,
both within and out of the context of a view
2016-04-13 15:10:24 +02:00
5ec2d8842e Character string indexer API 2016-04-12 17:21:01 +02:00
04c9470f7d Fixed and cleaned DNA_seq_indexer API 2016-04-12 17:20:24 +02:00
be05c889e2 DNA_seq_indexer API 2016-04-12 16:38:47 +02:00
04e3a7b5a9 Added more references in cython .cfiles files because it seems necessary
for linux distributions
2016-04-12 15:10:54 +02:00
d8107533d8 Obiblob_indexer API 2016-04-12 14:53:33 +02:00
cd4e65e190 Fixed typo and includes in obiblob files 2016-04-12 14:52:27 +02:00
375bfcce8a Renamed "Obi_byte_arrays" to "Obiblobs" and moved Obiblob functions to
separate obiblob.c and obiblob.h files
2016-04-12 11:21:14 +02:00
c225cfd8b6 Fixed bug with retrieval of values from AVLs (bad cast in byte array
structure)
2016-04-11 17:07:22 +02:00
6fe4c6134a Allows for calling getConfiguration without parametter for geting the
default configuration
2016-04-11 13:31:09 +02:00
966b1325ed Deleted declaration of obsolete public function 2016-04-11 11:14:20 +02:00
019dfc01b4 Branch to refactor and debug (AVLs bugged) 2016-04-08 15:38:57 +02:00
45c9c5075c A first version of the fasta parser 2016-04-01 18:15:54 +02:00
20b97c972b Add boolean type in the tag evaluation 2016-04-01 13:42:24 +02:00
efc4a4a3c6 Reduce the call count to eval. This reduce by 3 the time of fast(q|a)
header processing
2016-04-01 08:54:06 +02:00
ce6ea89c21 Add the missing bootstrappip module 2016-03-31 17:28:03 +02:00
4207db7c17 Transfers bug patch from orgasm 2016-03-31 16:53:09 +02:00
1cd35b3359 firt version of a fastq parser 2016-03-31 10:47:12 +02:00
f51a6df5b2 Add a class buffering lines during a text file reading 2016-03-30 14:53:25 +02:00
94417e1330 patch the uncompress module to be able to deal with remote file 2016-03-29 20:57:39 +02:00
2e17dbce55 Adds a uopen function able to open transparently a local or a remote
file compressed or not
2016-03-29 20:56:54 +02:00
a9eed1f5d9 Adds class for uncompressing transparently compressed files on line 2016-03-29 18:21:04 +02:00
2dfab3f378 Some changes in relation with the new obitools3.apps module 2016-03-28 15:05:59 +02:00
e583098a96 change in the obi programme according to the new obitools3.apps module
creation
2016-03-28 15:05:02 +02:00
b926ca3997 A template for a command 2016-03-28 15:04:06 +02:00
aacfefad26 A set of utilitaty function for creating commands 2016-03-28 15:03:26 +02:00
edc4fd7b3e Fixed minor warning 2016-03-25 16:11:52 +01:00
ff6c27acf2 Implemented the retrieval of values with groups of AVLs 2016-03-25 15:35:16 +01:00
69856f18dd untested (and no possible retrieval) of CRC used to represent data in
AVL trees
2016-03-24 16:38:11 +01:00
2c084c8cf7 Switch to 10000000 per avl 2016-03-23 16:13:28 +01:00
58ac860cc7 Added macro for the bloom filter parameters and deleted old unused
macros for crc
2016-03-23 13:33:40 +01:00
d44117d625 obiimport function for testing purposes 2016-03-23 13:00:02 +01:00
6bd42132c4 Minor fixes to silence warnings and replaced two asprintf uses 2016-03-23 12:58:53 +01:00
4085904362 Merge branch 'multiple_avls_bloom' 2016-03-22 14:14:10 +01:00
b04b4b5902 made POSIX compliant 2016-03-21 11:33:06 +01:00
383e738ab7 Merge branch 'master' of git@git.metabarcoding.org:obitools/obitools3.git 2016-03-18 15:49:53 +01:00
3681cecb4d Multiple AVLs with bloom filters (very raw test version) 2016-03-18 11:06:02 +01:00
545ed8111a Code for tests storing data in multiple AVLs.
(note: unretrievable data as implemented)
2016-03-11 15:34:55 +01:00
86071d30c9 Minor improvement in AVL initial size calculation 2016-03-11 14:07:40 +01:00
21d1b2ed3e First implementation of taxonomy reading 2016-03-11 13:56:38 +01:00
6157633137 prototype for the obi unix command and the count sub command 2016-03-08 16:06:00 +01:00
a08def47e6 It is now impossible to create a view with a name identical to one of an
existing written view
2016-03-01 13:36:54 +01:00
fc5a12bad7 Closes #34 2016-02-29 17:56:55 +01:00
e323d8e702 Cython classes for nucleotide sequences (outside or in the context of a
view)
2016-02-29 16:33:30 +01:00
b350ea0393 Fixed minor error 2016-02-29 16:28:34 +01:00
8e9e21a02e Increased the maximum depth of AVL trees 2016-02-29 16:27:23 +01:00
4df313c54a Added Obiviews specialized for the handling of nucleotide sequences 2016-02-25 09:43:27 +01:00
ffc68d448f Deleted a forgotten print statement 2016-02-18 15:15:42 +01:00
a8f03248a8 Major update : views 2016-02-18 10:38:51 +01:00
cfaf069095 Fixed more typos and formatting imperfections. 2015-12-11 17:37:25 +01:00
a6144eabe2 Fixed typos 2015-12-11 17:26:20 +01:00
c139367555 DNA sequences and character strings are now handled using AVL trees. 2015-12-11 17:24:44 +01:00
1586956d57 Added the lists of opened columns and arrays in the OBIDMS structure,
and a counter in the OBIDMS column structure; fixed some bugs and
created tests for referring columns that are bound to disappear anyway.
2015-12-02 17:32:07 +01:00
b45b496b0e Major update: new type of columns containing indices referring to lines
in other columns
2015-11-29 11:57:07 +01:00
2cf10cb6f0 Column type is now passed as a character string when creating the column
(either 'OBI_INT', 'OBI_FLOAT', 'OBI_BOOL', 'OBI_CHAR', 'OBI_STR' or
'OBI_SEQ')
2015-11-23 15:48:27 +01:00
5a5516303d deleting useless .pyc files 2015-11-23 14:43:34 +01:00
d6a99bafea Fixed a major bug with the versioning of columns that was introduced in
f6ec8ba9
2015-11-23 13:34:51 +01:00
08f2657e18 Increased maximum line count of columns to 1^9 2015-11-23 13:23:18 +01:00
6aa2f92930 DNA sequences are now encoded on 4 bits when they are in IUPAC 2015-11-20 15:32:09 +01:00
87044b41d8 modified the encoding function on 2 bits a little 2015-11-20 11:32:47 +01:00
6ab1c83302 New column type for DNA sequences. Only for those coded on 2 bits (only
'ATGCatgc') for now.
2015-11-19 18:12:48 +01:00
e371248567 changed version to 0.0.0 2015-11-19 18:11:21 +01:00
dbf9463238 The endianness of a DMS is now stored in the OBIDMS structure 2015-11-18 15:35:09 +01:00
eb12af4da4 Fixed minor error in the documentation of a function. 2015-11-16 15:38:01 +01:00
e8417b4f6f The endianness of an OBIDMS is now stored in an informations file that
is read when opening the OBIDMS.
2015-11-16 14:37:51 +01:00
6579566c6e Minor changes in code to improve readability and fix C compilation
warnings
2015-11-10 14:37:58 +01:00
410e2e02a0 When retrieving the header of a column, the version number of the column
wanted can now be provided.
2015-11-10 13:30:10 +01:00
8ce4f264aa When enlarging a column, the function doesn't try anymore to keep the
mapped region at the same pointer (never works), and unmap/remap
instead.
2015-11-10 13:18:36 +01:00
d885eb48ff The header size when creating a column is now calculated according to
the size of the header structure and the page size of the platform.
2015-11-10 13:09:30 +01:00
661fe3606a In OBI_CHAR columns, characters are now given and retrieved as decoded
(unicode) characters.
2015-11-10 11:24:08 +01:00
c4b7e579cf Comments in column headers are now working. 2015-11-10 10:56:45 +01:00
f6ec8ba963 The header size is now directly read in the file when a column or an
array is opened.
2015-11-09 17:50:32 +01:00
0e3d6ed2d7 Methods __len__ (number of lines used) and __sizeof__ (total size in
bytes) implemented for columns.
2015-11-09 15:56:20 +01:00
01bfc14503 The data size in bytes is now stored in the header of a column. 2015-11-09 15:55:00 +01:00
65c1b1e8b2 Minor changes to make the creation of files and directories cleaner 2015-11-09 15:22:01 +01:00
b37bd8f21c File descriptors for dms, column and array directories are now stored in
structures.
2015-11-09 15:06:02 +01:00
05e3956a0c Minor changes in code to improve readability (freeing some character
strings earlier)
2015-11-09 11:22:51 +01:00
9b066f4327 Major update: obiarrays with columns containing indices referring to
character strings.
2015-11-06 17:55:15 +01:00
456551ffeb obi arrays that don't work because of cython bug passing wrong pointers 2015-11-03 14:22:00 +01:00
ecb9d97adb Reorganized the code to have less functions, and the functions to get
and format the creation date of a column are now working.
2015-10-15 15:12:45 +02:00
0eaa5aa784 Major changes : new cython subclasses to handle columns with multiple
elements per line in a more efficient way + now elements_names are
passed as a list + new function to recover only the header of a column
2015-10-14 18:05:34 +02:00
21923e213d The unit tests now test for None values 2015-10-12 18:02:40 +02:00
6877fc4892 Fixed a critical bug where values were initialized to NA at the wrong
location when there was multiple elements per line
2015-10-12 17:54:36 +02:00
dbed3d9d1d New module for unit testing with PyUnit 2015-10-09 15:42:57 +02:00
fc8bf16769 Fixed a critical bug in the computation of the new number of lines of a
column when truncating
2015-10-09 13:49:48 +02:00
e114a3c9cb fixed a critical bug where data size was not calculated correctly and
column directory is now closed when column is closed
2015-10-09 10:25:40 +02:00
ebc9f6f512 fixed a bug where Cython was casting doubles in floats 2015-10-08 15:28:30 +02:00
2b3f03ec28 Removed deprecated script 2015-10-08 10:46:46 +02:00
8fd9c06be2 Fixed missing file for documentation compilation 2015-10-08 10:45:54 +02:00
b553eef781 Method to close a DMS is uncommented but not complete yet (columns have
to be closed separately)
2015-10-08 10:44:13 +02:00
ee4c513fd4 Fixed a bug where cloning a column would fail if the data was empty 2015-10-08 10:36:02 +02:00
c013e6ad33 fixed typo in doxygen doc 2015-10-08 10:33:19 +02:00
c98d567e2f Updated the documentation and restructured a bit because it wasn't
compiling (note: Breathe not working)
2015-10-06 11:09:01 +02:00
392f110c8d new functions in the OBIDMS_column class to raise NotImplementedError
exceptions and to get the creation date of a column
2015-10-02 13:51:26 +02:00
6ced3c4896 new functions to get the creation date of a column 2015-10-02 13:47:53 +02:00
4b8bf41a71 closes #13, obi_errno is initialized to 0 2015-10-02 13:46:34 +02:00
c59a244e9d Fixed little typo 2015-09-30 12:07:13 +02:00
4b7f2d268b Doxygen documentation corrected and completed. 2015-09-30 12:03:46 +02:00
45af8396b8 minor change in variable name for better coherence 2015-09-28 13:52:41 +02:00
eeebbc77c3 missing __init__ file in previous commit 2015-09-28 13:52:13 +02:00
72155c3b73 entirely reworked cython wrapper 2015-09-28 13:51:35 +02:00
9aaf2d264a test 2015-09-25 11:29:17 +02:00
0951934f77 test 2015-09-25 11:28:23 +02:00
68b6ab3224 working on the cython wrapper. This doesn't compile 2015-09-24 11:32:09 +02:00
b0570ee486 fixed permissions when creating files 2015-09-22 15:52:07 +02:00
72105b2aed fixed the computation of the data size when cloning a column 2015-09-22 10:27:17 +02:00
9ab8a42340 fixed the mkdir permission mode that forced the user to use sudo to
write in the created directories
2015-09-21 15:48:02 +02:00
7b606c0477 Column files now always have a size that is a multiple of the page size,
and the function that enlarges mapped column files tries mapping on next
byte before unmapping/mapping again
2015-09-21 15:42:29 +02:00
7d7dbb1bf9 Handling of single char columns 2015-09-18 15:49:15 +02:00
34a3717fb9 The cython function to open a column is now a method of the OBIDMS class 2015-09-17 11:23:50 +02:00
2698022aaf New OBIDMS method to list the columns of an OBIDMS 2015-09-15 17:09:31 +02:00
90bf15186e Method to get a column's number of lines used 2015-09-14 17:33:40 +02:00
5c3bc03bd2 Added the handling of errors with the functions to get a value in a
column
2015-09-14 17:04:29 +02:00
0e50fbf706 Merge branch 'master' of git@git.metabarcoding.org:obitools/obitools3.git 2015-09-03 08:38:57 -04:00
f2231770f1 If an error occurs when cloning a column, the new cloned file is now
deleted
2015-09-02 13:06:21 +02:00
f5937ec019 Deplacement des sources de la doc... 2015-09-02 04:50:50 -04:00
52e3f2ce4a Columns that are cloned are now closed after cloning 2015-09-02 10:36:00 +02:00
f0dffaff13 Merge branch 'master' of git@git.metabarcoding.org:obitools/obitools3.git 2015-09-02 10:15:18 +02:00
f31d8983bb Columns are now enlarged if needed when setting a value 2015-09-01 17:38:08 +02:00
a6395ebaf2 Transfert the distutil.ext from the org.asm project 2015-08-27 16:12:19 -04:00
3b7536c0df The number of lines used is updated in the header when a value is set,
and the iteration on the values of a column is done on the number of
lines used (instead of on the total line count).
2015-08-26 17:25:15 +02:00
34ade161de added an obidebug() line 2015-08-26 17:05:37 +02:00
48d10ea17c Functions to truncate and/or close a column 2015-08-26 17:01:54 +02:00
fdc3b96beb Added cython subclasses for columns with _writable or _read for each
data type. Trying to set a value on a _read subclass raises an
exception.
2015-08-26 15:34:12 +02:00
95dbeb25a0 Added functions to get and set values in columns using the element name
(for columns with lines made of vectors of elements), for all data types
2015-08-26 15:05:23 +02:00
9d91e907e5 When a column is created, its data is initialized to the NA value of the
obitype
2015-08-26 12:00:38 +02:00
3f81dc0173 Changed the variable name 'nb_elements' to 'nb_lines' for better clarity 2015-08-26 10:52:19 +02:00
8ff3488926 Fix: The number of lines used is set when cloning a column with its data 2015-08-26 10:38:07 +02:00
611699252e Files and functions for columns with the data type OBIChar_t, working
but using char* (needs to be changed)
2015-08-26 10:31:56 +02:00
9ad31fddff Added the possibility to clone a column, with or without its data 2015-08-26 10:29:07 +02:00
e6d96d999a Activated the html annotated files output in setup.py 2015-08-12 15:47:13 +02:00
eaca9c85f1 Files and functions for OBIDMS columns with the data type OBI_BOOL 2015-08-11 11:00:59 +02:00
bc2ca89088 Basis for OBIDMS columns with the type OBI_IDX 2015-08-10 16:30:55 +02:00
a5ed8f0ef0 A basic obicount for testing purposes 2015-08-10 16:06:05 +02:00
3b3267db9a Removed a useless #include 2015-08-10 16:05:34 +02:00
d53c16cf3e Functions and files for OBIDMS columns with the type OBI_FLOAT 2015-08-10 16:04:53 +02:00
66f397239b Commented __del__ functions for now, as they contain closedir() that can
create problems
2015-08-03 16:15:26 +02:00
1fe2e36d5d Fixed a critical bug where the elements names were not stored correctly 2015-08-03 16:11:55 +02:00
ab44e32afb More efficient obi_column_set_int() and obi_column_get_int() 2015-08-03 15:26:28 +02:00
41f627091f Error handling: obidebug() with message for all errors, and removed
commands closing directories when an error occurred (creating more
errors).
2015-08-03 15:10:39 +02:00
1e01c9059c functions to get data type as a character string 2015-08-03 11:46:21 +02:00
5f62cd8526 C sources modified to add the handling of OBIDM columns with the type
OBI_INT, and the handling of multiple types in general
2015-07-31 18:03:48 +02:00
a6abc74500 Cython sources modified to add the handling of OBIDMS columns with the
type OBI_INT
2015-07-31 18:02:40 +02:00
5c674715ee added the idea of each line of a column corresponding to a vector, with
the added informations of the number of elements per line and the
elements' names in the column's header structure
2015-07-20 16:08:50 +02:00
484fcca557 changed the private *at functions to the official ones for openat and
mkdirat, but not for opendirat, as there is no official version of it.
2015-07-20 11:38:43 +02:00
802f9983c2 fixing file names in list of C files after changing 'column groups' to
'column directories'
2015-07-20 11:13:56 +02:00
d26abc438d fixing an 'error' I had introduced to test the debugging system 2015-07-20 11:12:48 +02:00
2162bf4272 obidebug.h for debugging tools 2015-06-26 17:56:15 +02:00
c454f9612e changed 'column groups' to be called 'column directories' for now. 2015-06-26 17:53:03 +02:00
71cd59f775 close file descriptors that don't refer to a directory (as those can not
be reopened)
2015-06-24 13:53:12 +02:00
719ef61461 changed file names 2015-06-23 18:36:20 +02:00
152b34b5f4 Closes #10 : column groups stored in directories 2015-06-23 18:35:34 +02:00
61b6c3ce83 Added OBI errnos but work in progress 2015-06-23 17:56:21 +02:00
0a78a786c1 fixed replacement *at functions and made get_full_path() function public 2015-06-23 17:55:00 +02:00
e6e68786f7 private *at functions (openat, mkdirat, opendirat, closedirat) 2015-06-18 17:19:23 +02:00
58e7066b9f updated and fixed the documentation 2015-06-17 17:17:54 +02:00
ed662077b3 fixed bugs and typos in the obidms and obidmscolumn source and header
files
2015-06-17 16:51:16 +02:00
9f9c20aa4a new OBIDMS_column cython class 2015-06-17 16:49:49 +02:00
73ba9fd4c3 updated the OBIDMS class 2015-06-17 16:48:56 +02:00
771be89103 updated the documentation (replaced 'release files' with 'version
files')
2015-06-17 16:45:04 +02:00
f740557d96 fixes #9 : replacement function for openat(). 2015-06-17 14:55:05 +02:00
aff74c72dd updated the documentation 2015-06-10 15:41:14 +02:00
5d967f5b7b fixed typos and other minor things 2015-06-10 15:19:02 +02:00
9b6de9c411 Attempt at defining some naming conventions. 2015-05-28 17:54:11 +02:00
27e246e709 updated the documentation 2015-05-28 17:03:50 +02:00
ff64d1b2e9 deleting a useless file 2015-05-28 16:35:43 +02:00
235a81a534 deleting test data file 2015-05-28 16:35:03 +02:00
4759e1cf54 deleting the Breathe sources 2015-05-28 16:34:24 +02:00
f4a123cd17 updated the documentation with the special values, and the idea of
column directories and column group directories.
2015-05-28 16:33:45 +02:00
439 changed files with 65349 additions and 15057 deletions

518
LICENSE Executable file
View File

@ -0,0 +1,518 @@
CeCILL FREE SOFTWARE LICENSE AGREEMENT
Version 2.1 dated 2013-06-21
Notice
This Agreement is a Free Software license agreement that is the result
of discussions between its authors in order to ensure compliance with
the two main principles guiding its drafting:
* firstly, compliance with the principles governing the distribution
of Free Software: access to source code, broad rights granted to users,
* secondly, the election of a governing law, French law, with which it
is conformant, both as regards the law of torts and intellectual
property law, and the protection that it offers to both authors and
holders of the economic rights over software.
The authors of the CeCILL (for Ce[a] C[nrs] I[nria] L[ogiciel] L[ibre])
license are:
Commissariat à l'énergie atomique et aux énergies alternatives - CEA, a
public scientific, technical and industrial research establishment,
having its principal place of business at 25 rue Leblanc, immeuble Le
Ponant D, 75015 Paris, France.
Centre National de la Recherche Scientifique - CNRS, a public scientific
and technological establishment, having its principal place of business
at 3 rue Michel-Ange, 75794 Paris cedex 16, France.
Institut National de Recherche en Informatique et en Automatique -
Inria, a public scientific and technological establishment, having its
principal place of business at Domaine de Voluceau, Rocquencourt, BP
105, 78153 Le Chesnay cedex, France.
Preamble
The purpose of this Free Software license agreement is to grant users
the right to modify and redistribute the software governed by this
license within the framework of an open source distribution model.
The exercising of this right is conditional upon certain obligations for
users so as to preserve this status for all subsequent redistributions.
In consideration of access to the source code and the rights to copy,
modify and redistribute granted by the license, users are provided only
with a limited warranty and the software's author, the holder of the
economic rights, and the successive licensors only have limited liability.
In this respect, the risks associated with loading, using, modifying
and/or developing or reproducing the software by the user are brought to
the user's attention, given its Free Software status, which may make it
complicated to use, with the result that its use is reserved for
developers and experienced professionals having in-depth computer
knowledge. Users are therefore encouraged to load and test the
suitability of the software as regards their requirements in conditions
enabling the security of their systems and/or data to be ensured and,
more generally, to use and operate it in the same conditions of
security. This Agreement may be freely reproduced and published,
provided it is not altered, and that no provisions are either added or
removed herefrom.
This Agreement may apply to any or all software for which the holder of
the economic rights decides to submit the use thereof to its provisions.
Frequently asked questions can be found on the official website of the
CeCILL licenses family (http://www.cecill.info/index.en.html) for any
necessary clarification.
Article 1 - DEFINITIONS
For the purpose of this Agreement, when the following expressions
commence with a capital letter, they shall have the following meaning:
Agreement: means this license agreement, and its possible subsequent
versions and annexes.
Software: means the software in its Object Code and/or Source Code form
and, where applicable, its documentation, "as is" when the Licensee
accepts the Agreement.
Initial Software: means the Software in its Source Code and possibly its
Object Code form and, where applicable, its documentation, "as is" when
it is first distributed under the terms and conditions of the Agreement.
Modified Software: means the Software modified by at least one
Contribution.
Source Code: means all the Software's instructions and program lines to
which access is required so as to modify the Software.
Object Code: means the binary files originating from the compilation of
the Source Code.
Holder: means the holder(s) of the economic rights over the Initial
Software.
Licensee: means the Software user(s) having accepted the Agreement.
Contributor: means a Licensee having made at least one Contribution.
Licensor: means the Holder, or any other individual or legal entity, who
distributes the Software under the Agreement.
Contribution: means any or all modifications, corrections, translations,
adaptations and/or new functions integrated into the Software by any or
all Contributors, as well as any or all Internal Modules.
Module: means a set of sources files including their documentation that
enables supplementary functions or services in addition to those offered
by the Software.
External Module: means any or all Modules, not derived from the
Software, so that this Module and the Software run in separate address
spaces, with one calling the other when they are run.
Internal Module: means any or all Module, connected to the Software so
that they both execute in the same address space.
GNU GPL: means the GNU General Public License version 2 or any
subsequent version, as published by the Free Software Foundation Inc.
GNU Affero GPL: means the GNU Affero General Public License version 3 or
any subsequent version, as published by the Free Software Foundation Inc.
EUPL: means the European Union Public License version 1.1 or any
subsequent version, as published by the European Commission.
Parties: mean both the Licensee and the Licensor.
These expressions may be used both in singular and plural form.
Article 2 - PURPOSE
The purpose of the Agreement is the grant by the Licensor to the
Licensee of a non-exclusive, transferable and worldwide license for the
Software as set forth in Article 5 <#scope> hereinafter for the whole
term of the protection granted by the rights over said Software.
Article 3 - ACCEPTANCE
3.1 The Licensee shall be deemed as having accepted the terms and
conditions of this Agreement upon the occurrence of the first of the
following events:
* (i) loading the Software by any or all means, notably, by
downloading from a remote server, or by loading from a physical medium;
* (ii) the first time the Licensee exercises any of the rights granted
hereunder.
3.2 One copy of the Agreement, containing a notice relating to the
characteristics of the Software, to the limited warranty, and to the
fact that its use is restricted to experienced users has been provided
to the Licensee prior to its acceptance as set forth in Article 3.1
<#accepting> hereinabove, and the Licensee hereby acknowledges that it
has read and understood it.
Article 4 - EFFECTIVE DATE AND TERM
4.1 EFFECTIVE DATE
The Agreement shall become effective on the date when it is accepted by
the Licensee as set forth in Article 3.1 <#accepting>.
4.2 TERM
The Agreement shall remain in force for the entire legal term of
protection of the economic rights over the Software.
Article 5 - SCOPE OF RIGHTS GRANTED
The Licensor hereby grants to the Licensee, who accepts, the following
rights over the Software for any or all use, and for the term of the
Agreement, on the basis of the terms and conditions set forth hereinafter.
Besides, if the Licensor owns or comes to own one or more patents
protecting all or part of the functions of the Software or of its
components, the Licensor undertakes not to enforce the rights granted by
these patents against successive Licensees using, exploiting or
modifying the Software. If these patents are transferred, the Licensor
undertakes to have the transferees subscribe to the obligations set
forth in this paragraph.
5.1 RIGHT OF USE
The Licensee is authorized to use the Software, without any limitation
as to its fields of application, with it being hereinafter specified
that this comprises:
1. permanent or temporary reproduction of all or part of the Software
by any or all means and in any or all form.
2. loading, displaying, running, or storing the Software on any or all
medium.
3. entitlement to observe, study or test its operation so as to
determine the ideas and principles behind any or all constituent
elements of said Software. This shall apply when the Licensee
carries out any or all loading, displaying, running, transmission or
storage operation as regards the Software, that it is entitled to
carry out hereunder.
5.2 ENTITLEMENT TO MAKE CONTRIBUTIONS
The right to make Contributions includes the right to translate, adapt,
arrange, or make any or all modifications to the Software, and the right
to reproduce the resulting software.
The Licensee is authorized to make any or all Contributions to the
Software provided that it includes an explicit notice that it is the
author of said Contribution and indicates the date of the creation thereof.
5.3 RIGHT OF DISTRIBUTION
In particular, the right of distribution includes the right to publish,
transmit and communicate the Software to the general public on any or
all medium, and by any or all means, and the right to market, either in
consideration of a fee, or free of charge, one or more copies of the
Software by any means.
The Licensee is further authorized to distribute copies of the modified
or unmodified Software to third parties according to the terms and
conditions set forth hereinafter.
5.3.1 DISTRIBUTION OF SOFTWARE WITHOUT MODIFICATION
The Licensee is authorized to distribute true copies of the Software in
Source Code or Object Code form, provided that said distribution
complies with all the provisions of the Agreement and is accompanied by:
1. a copy of the Agreement,
2. a notice relating to the limitation of both the Licensor's warranty
and liability as set forth in Articles 8 and 9,
and that, in the event that only the Object Code of the Software is
redistributed, the Licensee allows effective access to the full Source
Code of the Software for a period of at least three years from the
distribution of the Software, it being understood that the additional
acquisition cost of the Source Code shall not exceed the cost of the
data transfer.
5.3.2 DISTRIBUTION OF MODIFIED SOFTWARE
When the Licensee makes a Contribution to the Software, the terms and
conditions for the distribution of the resulting Modified Software
become subject to all the provisions of this Agreement.
The Licensee is authorized to distribute the Modified Software, in
source code or object code form, provided that said distribution
complies with all the provisions of the Agreement and is accompanied by:
1. a copy of the Agreement,
2. a notice relating to the limitation of both the Licensor's warranty
and liability as set forth in Articles 8 and 9,
and, in the event that only the object code of the Modified Software is
redistributed,
3. a note stating the conditions of effective access to the full source
code of the Modified Software for a period of at least three years
from the distribution of the Modified Software, it being understood
that the additional acquisition cost of the source code shall not
exceed the cost of the data transfer.
5.3.3 DISTRIBUTION OF EXTERNAL MODULES
When the Licensee has developed an External Module, the terms and
conditions of this Agreement do not apply to said External Module, that
may be distributed under a separate license agreement.
5.3.4 COMPATIBILITY WITH OTHER LICENSES
The Licensee can include a code that is subject to the provisions of one
of the versions of the GNU GPL, GNU Affero GPL and/or EUPL in the
Modified or unmodified Software, and distribute that entire code under
the terms of the same version of the GNU GPL, GNU Affero GPL and/or EUPL.
The Licensee can include the Modified or unmodified Software in a code
that is subject to the provisions of one of the versions of the GNU GPL,
GNU Affero GPL and/or EUPL and distribute that entire code under the
terms of the same version of the GNU GPL, GNU Affero GPL and/or EUPL.
Article 6 - INTELLECTUAL PROPERTY
6.1 OVER THE INITIAL SOFTWARE
The Holder owns the economic rights over the Initial Software. Any or
all use of the Initial Software is subject to compliance with the terms
and conditions under which the Holder has elected to distribute its work
and no one shall be entitled to modify the terms and conditions for the
distribution of said Initial Software.
The Holder undertakes that the Initial Software will remain ruled at
least by this Agreement, for the duration set forth in Article 4.2 <#term>.
6.2 OVER THE CONTRIBUTIONS
The Licensee who develops a Contribution is the owner of the
intellectual property rights over this Contribution as defined by
applicable law.
6.3 OVER THE EXTERNAL MODULES
The Licensee who develops an External Module is the owner of the
intellectual property rights over this External Module as defined by
applicable law and is free to choose the type of agreement that shall
govern its distribution.
6.4 JOINT PROVISIONS
The Licensee expressly undertakes:
1. not to remove, or modify, in any manner, the intellectual property
notices attached to the Software;
2. to reproduce said notices, in an identical manner, in the copies of
the Software modified or not.
The Licensee undertakes not to directly or indirectly infringe the
intellectual property rights on the Software of the Holder and/or
Contributors, and to take, where applicable, vis-à-vis its staff, any
and all measures required to ensure respect of said intellectual
property rights of the Holder and/or Contributors.
Article 7 - RELATED SERVICES
7.1 Under no circumstances shall the Agreement oblige the Licensor to
provide technical assistance or maintenance services for the Software.
However, the Licensor is entitled to offer this type of services. The
terms and conditions of such technical assistance, and/or such
maintenance, shall be set forth in a separate instrument. Only the
Licensor offering said maintenance and/or technical assistance services
shall incur liability therefor.
7.2 Similarly, any Licensor is entitled to offer to its licensees, under
its sole responsibility, a warranty, that shall only be binding upon
itself, for the redistribution of the Software and/or the Modified
Software, under terms and conditions that it is free to decide. Said
warranty, and the financial terms and conditions of its application,
shall be subject of a separate instrument executed between the Licensor
and the Licensee.
Article 8 - LIABILITY
8.1 Subject to the provisions of Article 8.2, the Licensee shall be
entitled to claim compensation for any direct loss it may have suffered
from the Software as a result of a fault on the part of the relevant
Licensor, subject to providing evidence thereof.
8.2 The Licensor's liability is limited to the commitments made under
this Agreement and shall not be incurred as a result of in particular:
(i) loss due the Licensee's total or partial failure to fulfill its
obligations, (ii) direct or consequential loss that is suffered by the
Licensee due to the use or performance of the Software, and (iii) more
generally, any consequential loss. In particular the Parties expressly
agree that any or all pecuniary or business loss (i.e. loss of data,
loss of profits, operating loss, loss of customers or orders,
opportunity cost, any disturbance to business activities) or any or all
legal proceedings instituted against the Licensee by a third party,
shall constitute consequential loss and shall not provide entitlement to
any or all compensation from the Licensor.
Article 9 - WARRANTY
9.1 The Licensee acknowledges that the scientific and technical
state-of-the-art when the Software was distributed did not enable all
possible uses to be tested and verified, nor for the presence of
possible defects to be detected. In this respect, the Licensee's
attention has been drawn to the risks associated with loading, using,
modifying and/or developing and reproducing the Software which are
reserved for experienced users.
The Licensee shall be responsible for verifying, by any or all means,
the suitability of the product for its requirements, its good working
order, and for ensuring that it shall not cause damage to either persons
or properties.
9.2 The Licensor hereby represents, in good faith, that it is entitled
to grant all the rights over the Software (including in particular the
rights set forth in Article 5 <#scope>).
9.3 The Licensee acknowledges that the Software is supplied "as is" by
the Licensor without any other express or tacit warranty, other than
that provided for in Article 9.2 <#good-faith> and, in particular,
without any warranty as to its commercial value, its secured, safe,
innovative or relevant nature.
Specifically, the Licensor does not warrant that the Software is free
from any error, that it will operate without interruption, that it will
be compatible with the Licensee's own equipment and software
configuration, nor that it will meet the Licensee's requirements.
9.4 The Licensor does not either expressly or tacitly warrant that the
Software does not infringe any third party intellectual property right
relating to a patent, software or any other property right. Therefore,
the Licensor disclaims any and all liability towards the Licensee
arising out of any or all proceedings for infringement that may be
instituted in respect of the use, modification and redistribution of the
Software. Nevertheless, should such proceedings be instituted against
the Licensee, the Licensor shall provide it with technical and legal
expertise for its defense. Such technical and legal expertise shall be
decided on a case-by-case basis between the relevant Licensor and the
Licensee pursuant to a memorandum of understanding. The Licensor
disclaims any and all liability as regards the Licensee's use of the
name of the Software. No warranty is given as regards the existence of
prior rights over the name of the Software or as regards the existence
of a trademark.
Article 10 - TERMINATION
10.1 In the event of a breach by the Licensee of its obligations
hereunder, the Licensor may automatically terminate this Agreement
thirty (30) days after notice has been sent to the Licensee and has
remained ineffective.
10.2 A Licensee whose Agreement is terminated shall no longer be
authorized to use, modify or distribute the Software. However, any
licenses that it may have granted prior to termination of the Agreement
shall remain valid subject to their having been granted in compliance
with the terms and conditions hereof.
Article 11 - MISCELLANEOUS
11.1 EXCUSABLE EVENTS
Neither Party shall be liable for any or all delay, or failure to
perform the Agreement, that may be attributable to an event of force
majeure, an act of God or an outside cause, such as defective
functioning or interruptions of the electricity or telecommunications
networks, network paralysis following a virus attack, intervention by
government authorities, natural disasters, water damage, earthquakes,
fire, explosions, strikes and labor unrest, war, etc.
11.2 Any failure by either Party, on one or more occasions, to invoke
one or more of the provisions hereof, shall under no circumstances be
interpreted as being a waiver by the interested Party of its right to
invoke said provision(s) subsequently.
11.3 The Agreement cancels and replaces any or all previous agreements,
whether written or oral, between the Parties and having the same
purpose, and constitutes the entirety of the agreement between said
Parties concerning said purpose. No supplement or modification to the
terms and conditions hereof shall be effective as between the Parties
unless it is made in writing and signed by their duly authorized
representatives.
11.4 In the event that one or more of the provisions hereof were to
conflict with a current or future applicable act or legislative text,
said act or legislative text shall prevail, and the Parties shall make
the necessary amendments so as to comply with said act or legislative
text. All other provisions shall remain effective. Similarly, invalidity
of a provision of the Agreement, for any reason whatsoever, shall not
cause the Agreement as a whole to be invalid.
11.5 LANGUAGE
The Agreement is drafted in both French and English and both versions
are deemed authentic.
Article 12 - NEW VERSIONS OF THE AGREEMENT
12.1 Any person is authorized to duplicate and distribute copies of this
Agreement.
12.2 So as to ensure coherence, the wording of this Agreement is
protected and may only be modified by the authors of the License, who
reserve the right to periodically publish updates or new versions of the
Agreement, each with a separate number. These subsequent versions may
address new issues encountered by Free Software.
12.3 Any Software distributed under a given version of the Agreement may
only be subsequently distributed under the same version of the Agreement
or a subsequent version, subject to the provisions of Article 5.3.4
<#compatibility>.
Article 13 - GOVERNING LAW AND JURISDICTION
13.1 The Agreement is governed by French law. The Parties agree to
endeavor to seek an amicable solution to any disagreements or disputes
that may arise during the performance of the Agreement.
13.2 Failing an amicable solution within two (2) months as from their
occurrence, and unless emergency proceedings are necessary, the
disagreements or disputes shall be referred to the Paris Courts having
jurisdiction, by the more diligent Party.

0
MANIFEST.in Normal file → Executable file
View File

40
README.md Executable file
View File

@ -0,0 +1,40 @@
The `OBITools3`: A package for the management of analyses and data in DNA metabarcoding
---------------------------------------------
DNA metabarcoding offers new perspectives for biodiversity research [1]. This approach of ecosystem studies relies heavily on the use of Next-Generation Sequencing (NGS), and consequently requires the ability to to treat large volumes of data. The `OBITools` package satisfies this requirement thanks to a set of programs specifically designed for analyzing NGS data in a DNA metabarcoding context [2] - <http://metabarcoding.org/obitools>. Their capacity to filter and edit sequences while taking into account taxonomic annotation helps to setup tailored-made analysis pipelines for a broad range of DNA metabarcoding applications, including biodiversity surveys or diet analyses.
**The `OBITools3`.** This new version of the `OBITools` looks to significantly improve the storage efficiency and the data processing speed. To this end, the `OBITools3` rely on an ad hoc database system, inside which all the data that a DNA metabarcoding experiment must consider is stored: the sequences, the metadata (describing for instance the samples), the database containing the reference sequences used for the taxonomic annotation, as well as the taxonomic databases. Besides the gain in efficiency, this new structure allows an easier access to all the data associated with an experiment.
**Column-oriented storage.** An analysis pipeline corresponds to a succession of commands, each computing one step of the analysis, and where the result of the command *n* is used by the command *n+1*. DNA metabarcoding data can easily be represented in the form of tables, and each command can be regarded as an operation transforming one or several 'input' tables into one or several 'output' tables, which can be used by the next command. Many of the basic operations in a pipeline copy without modification an important part of the input tables to the result tables, and use for their calculations only a small part of the input data. In the original `OBITools`, those tables are kept in the form of annotated sequence files in the FASTA or FASTQ format. This has two consequences: i) keeping the transitional results of the analysis pipeline means using disk space for an important volume of redundant data, ii) The coding and decoding of informations that are not actually used represent an important part of the treatment process. The new database system used by the `OBITools3` (called DMS for Data Management System) relies on column-oriented storage. The columns are immutable and can be assembled in views representing the data tables. This way, the data not modified by a command in an input table can easily be associated to the result table without duplicating any information ; and the data not used at all by a command can be associated with the result table without being read. This strategy results in a gain in disk space efficiency by limiting data redundancy, as well as a gain in execution time by limiting data reading, writing and conversion operations. Finally, as a mean to optimize data access, each column is stored in a binary file directly mapped in memory for reading and writing operations.
**Storage optimization.** DNA metabarcoding data is intrinsically very redundant. For example, the same sequence corresponding to a species will be present several thousand times across all samples. In order to limit the disk space used and make comparison operations more efficient, data in the form of character strings is stored in columns using a complex indexing structure, efficient on millions of values, coupling hash functions, Bloom filters and AVL trees. Finally, DNA sequences are compressed by encoding each nucleotide on two or four bits depending on whether the sequences contain only the four nucleotides (A, C, G, T) or use the IUPAC codes.
**Saving the data processing history.** The totality of the informations used by the `OBITools3` is stored in immutable data structures in the DMS. If a command has to modify a column used as input to produce its result, a new version of that column is created, leaving the initial version intact. This storage system enables to keep, at minimal cost, the totality of the transitional results produced by the pipeline. The storage of metadata describing all the operations that have produced a view (a result table) in the DMS makes possible the creation of an oriented hypergraph, where each node corresponds to a view and each arrow to an operation. By retracing the dependency relationships in this hypergraph, it is possible to rebuild *a posteriori* the entirety of the process that has produced a result table.
**Tools.** The `OBITools3` offer the same tools as the original `OBITools`. Eventually, new versions of `ecoPrimers` (PCR primer design) [3], `ecoPCR` (*in silico* PCR) [4], as well as `Sumatra` (sequence alignment) and `Sumaclust` (sequence alignment and clustering) [5] will be added, taking advantage of the database structure developed for the `OBITools3`.
**Implementation and disponibility.** The lower layers managing the DMS as well as all the compute-intensive functions are coded in `C99` for efficiency reasons. A `Cython` (<http://www.cython.org>) object layer allows for a simple but efficient implementation of the `OBITools3` commands in `Python 3.5`. The `OBITools3` are still in development, and the first functional versions are expected for autumn 2016.
**References.**
1. Taberlet P, Coissac E, Hajibabaei M, Rieseberg LH: Environmental DNA. Mol Ecol 2012:17891793.
2. Boyer F, Mercier C, Bonin A, Le Bras Y, Taberlet P, Coissac E: OBITools: a Unix-inspired software package for DNA metabarcoding. Mol Ecol Resour 2015:n/an/a.
3. Riaz T, Shehzad W, Viari A, Pompanon F, Taberlet P, Coissac E: ecoPrimers: inference of new DNA barcode markers from whole genome sequence analysis. Nucleic Acids Res 2011, 39:e145.
4. Ficetola GF, Coissac E, Zundel S, Riaz T, Shehzad W, Bessière J, Taberlet P, Pompanon F: An in silico approach for the evaluation of DNA barcodes. BMC Genomics 2010, 11:434.
5. Mercier C, Boyer F, Bonin A, Coissac E (2013) SUMATRA and SUMACLUST: fast and exact comparison and clustering of sequences. Available: <http://metabarcoding.org/sumatra> and <http://metabarcoding.org/sumaclust>

0
c-sandbox/obicount/Makefile Normal file → Executable file
View File

0
c-sandbox/obicount/obicount.c Normal file → Executable file
View File

View File

@ -10,33 +10,29 @@ from obidistutils.serenity.checksystem import is_mac_system
class build(ori_build):
def has_ctools(self):
return self.distribution.has_ctools()
def has_files(self):
return self.distribution.has_files()
def has_executables(self):
return self.distribution.has_executables()
def has_ext_modules(self):
return self.distribution.has_ext_modules()
def has_littlebigman(self):
return True
def has_pidname(self):
return is_mac_system()
def has_doc(self):
return True
def has_littlebigman(self):
return True
sub_commands = [('littlebigman', has_littlebigman),
('pidname',has_pidname),
('build_ctools', has_ctools),
('build_files', has_files),
('build_cexe', has_executables)] \
+ ori_build.sub_commands + \
[('build_sphinx',has_doc)]
try:
from obidistutils.command.build_sphinx import build_sphinx # @UnusedImport
sub_commands = [("littlebigman",has_littlebigman),
('pidname',has_pidname)
] \
+ ori_build.sub_commands + \
[('build_sphinx',has_doc)]
except ImportError:
sub_commands = [("littlebigman",has_littlebigman),
('pidname',has_pidname)
] \
+ ori_build.sub_commands

View File

@ -4,10 +4,11 @@ Created on 20 oct. 2012
@author: coissac
'''
from obidistutils.command.build_ctools import build_ctools
from .build_ctools import build_ctools
from .build_exe import build_exe
from distutils.errors import DistutilsSetupError
from distutils import log
import os
class build_cexe(build_ctools):
@ -38,7 +39,9 @@ class build_cexe(build_ctools):
self.set_undefined_options('build_files',
('files', 'built_files'))
self.executables = self.distribution.executables
self.executables = self.distribution.executables
# self.build_cexe = os.path.join(os.path.dirname(self.build_cexe),'cbinaries')
# self.mkpath(self.build_cexe)
if self.executables:
self.check_executable_list(self.executables)
@ -70,4 +73,13 @@ class build_cexe(build_ctools):
log.info("%s ok",message)
return sources
def run(self):
for cmd_name in self.get_sub_commands():
self.run_command(cmd_name)
build_exe.run(self)

View File

@ -5,7 +5,8 @@ Created on 20 oct. 2012
'''
from obidistutils.command.build_exe import build_exe
from .build_exe import build_exe
from distutils import log
class build_ctools(build_exe):
description = "build C/C++ executable not distributed with Python extensions"
@ -37,19 +38,26 @@ class build_ctools(build_exe):
self.executables = self.distribution.ctools
self.check_executable_list(self.executables)
if self.littlebigman =='-DLITTLE_END':
if self.define is None:
self.define=[('LITTLE_END',None)]
else:
self.define.append('LITTLE_END',None)
log.info('Look for CPU architecture... %s',self.define)
self.ctools = set()
def run(self):
for cmd_name in self.get_sub_commands():
self.run_command(cmd_name)
build_exe.run(self)
for e,p in self.executables: # @UnusedVariable
self.ctools.add(e)

View File

@ -6,12 +6,28 @@ Created on 20 oct. 2012
import os
from distutils import sysconfig
from distutils.core import Command
from distutils.sysconfig import customize_compiler
from distutils.sysconfig import customize_compiler as customize_compiler_ori
from distutils.errors import DistutilsSetupError
from distutils import log
from distutils.ccompiler import show_compilers
def customize_compiler(compiler):
customize_compiler_ori(compiler)
compilername = compiler.compiler[0]
if ("gcc" in compilername or "g++" in compilername):
cc_cmd = ' '.join(compiler.compiler + ['-fopenmp'])
ccshared= ' '.join(x for x in sysconfig.get_config_vars("ccshared") if x is not None)
compiler.set_executables(
compiler=cc_cmd,
compiler_so=cc_cmd + ' ' + ccshared
)
class build_exe(Command):
description = "build an executable -- Abstract command "
@ -80,6 +96,7 @@ class build_exe(Command):
else:
self.extra_compile_args.append('-m%s' % self.sse)
# XXX same as for build_ext -- what about 'self.define' and
# 'self.undef' ?
@ -96,7 +113,7 @@ class build_exe(Command):
dry_run=self.dry_run,
force=self.force)
customize_compiler(self.compiler)
if self.include_dirs is not None:
self.compiler.set_include_dirs(self.include_dirs)
if self.define is not None:
@ -208,3 +225,4 @@ class build_exe(Command):
output_dir=self.build_cexe,
debug=self.debug)

View File

@ -7,87 +7,137 @@ Created on 13 fevr. 2014
from distutils import log
import os
from Cython.Distutils import build_ext as ori_build_ext # @UnresolvedImport
from distutils import sysconfig
from distutils.errors import DistutilsSetupError
class build_ext(ori_build_ext):
def modifyDocScripts(self):
build_dir_file=open("doc/build_dir.txt","w")
print(self.build_lib,file=build_dir_file)
build_dir_file.close()
def initialize_options(self):
ori_build_ext.initialize_options(self) # @UndefinedVariable
self.littlebigman = None
self.built_files = None
def finalize_options(self):
ori_build_ext.finalize_options(self) # @UndefinedVariable
self.set_undefined_options('littlebigman',
('littlebigman', 'littlebigman'))
self.set_undefined_options('build_files',
('files', 'built_files'))
self.cython_c_in_temp = 1
if self.littlebigman =='-DLITTLE_END':
if self.define is None:
self.define=[('LITTLE_END',None)]
else:
self.define.append('LITTLE_END',None)
def substitute_sources(self,exe_name,sources):
"""
Substitutes source file name starting by an @ by the actual
name of the built file (see --> build_files)
"""
sources = list(sources)
for i in range(len(sources)):
message = "%s :-> %s" % (exe_name,sources[i])
if sources[i][0]=='@':
try:
filename = self.built_files[sources[i][1:]]
except KeyError:
tmpfilename = os.path.join(self.build_temp,sources[i][1:])
if os.path.isfile (tmpfilename):
filename = tmpfilename
else:
raise DistutilsSetupError(
'The %s filename declared in the source '
'files of the program %s have not been '
'built by the installation process' % (sources[i],
exe_name))
sources[i]=filename
log.info("%s changed to %s",message,filename)
else:
log.info("%s ok",message)
return sources
def build_extensions(self):
# First, sanity-check the 'extensions' list
for ext in self.extensions:
ext.sources = self.substitute_sources(ext.name,ext.sources)
def _customize_compiler(compiler):
compilername = compiler.compiler[0]
if ("gcc" in compilername or "g++" in compilername):
cc_cmd = ' '.join(compiler.compiler + ['-fopenmp'])
ccshared= ' '.join(x for x in sysconfig.get_config_vars("ccshared") if x is not None)
self.check_extensions_list(self.extensions)
compiler.set_executables(
compiler=cc_cmd,
compiler_so=cc_cmd + ' ' + ccshared
)
for ext in self.extensions:
log.info("%s :-> %s",ext.name,ext.sources)
ext.sources = self.cython_sources(ext.sources, ext)
self.build_extension(ext)
try:
from Cython.Distutils import build_ext as ori_build_ext # @UnresolvedImport
from Cython.Compiler import Options as cython_options # @UnresolvedImport
class build_ext(ori_build_ext):
def run(self):
self.modifyDocScripts()
ori_build_ext.run(self) # @UndefinedVariable
def modifyDocScripts(self):
try:
os.mkdir("doc/sphinx")
except:
pass
build_dir_file=open("doc/sphinx/build_dir.txt","w")
print(self.build_lib,file=build_dir_file)
build_dir_file.close()
def initialize_options(self):
ori_build_ext.initialize_options(self) # @UndefinedVariable
self.littlebigman = None
self.built_files = None
def finalize_options(self):
super(build_ext, self).finalize_options()
self.set_undefined_options('littlebigman',
('littlebigman', 'littlebigman'))
self.set_undefined_options('build_files',
('files', 'built_files'))
self.cython_c_in_temp = 1
if self.littlebigman =='-DLITTLE_END':
if self.define is None:
self.define=[('LITTLE_END',None)]
else:
self.define.append('LITTLE_END',None)
def substitute_sources(self,exe_name,sources):
"""
Substitutes source file name starting by an @ by the actual
name of the built file (see --> build_files)
"""
sources = list(sources)
for i in range(len(sources)):
message = "%s :-> %s" % (exe_name,sources[i])
if sources[i][0]=='@':
try:
filename = self.built_files[sources[i][1:]]
except KeyError:
tmpfilename = os.path.join(self.build_temp,sources[i][1:])
if os.path.isfile (tmpfilename):
filename = tmpfilename
else:
raise DistutilsSetupError(
'The %s filename declared in the source '
'files of the program %s have not been '
'built by the installation process' % (sources[i],
exe_name))
sources[i]=filename
log.info("%s changed to %s",message,filename)
else:
log.info("%s ok",message)
return sources
def build_extensions(self):
# First, sanity-check the 'extensions' list
for ext in self.extensions:
ext.sources = self.substitute_sources(ext.name,ext.sources)
self.check_extensions_list(self.extensions)
print("pouic")
print(ext.sources)
print("pouac")
for ext in self.extensions:
log.info("%s :-> %s",ext.name,ext.sources)
ext.sources = self.cython_sources(ext.sources, ext)
self.build_extension(ext)
def build_extensions(self): # TODO what?? double? is it supposed to be build_extension?
if hasattr(self, 'compiler'):
_customize_compiler(self.compiler)
if hasattr(self, 'shlib_compiler'):
_customize_compiler(self.shlib_compiler)
ori_build_ext.build_extensions(self)
def run(self):
self.modifyDocScripts()
for cmd_name in self.get_sub_commands():
self.run_command(cmd_name)
cython_options.annotate = True
ori_build_ext.run(self) # @UndefinedVariable
def has_files(self):
return self.distribution.has_files()
def has_executables(self):
return self.distribution.has_executables()
sub_commands = [('build_files',has_files),
('build_cexe', has_executables)
] + ori_build_ext.sub_commands
except ImportError:
from distutils.command import build_ext # @UnusedImport

View File

@ -28,6 +28,10 @@ class build_files(Command):
self.files = {}
def run(self):
for cmd_name in self.get_sub_commands():
self.run_command(cmd_name)
for dest,prog,command in self.distribution.files:
destfile = os.path.join(self.build_temp,dest)
@ -48,6 +52,12 @@ class build_files(Command):
log.info("Done.\n")
def has_ctools(self):
return self.distribution.has_ctools()
sub_commands = [('build_ctools', has_ctools)] + \
Command.sub_commands

View File

@ -6,14 +6,14 @@ Created on 20 oct. 2012
import os.path
from distutils.command.build_scripts import build_scripts as ori_build_scripts,\
first_line_re
from distutils.command.build_scripts import build_scripts as ori_build_scripts
from distutils.util import convert_path
from distutils import log, sysconfig
from distutils.dep_util import newer
from stat import ST_MODE
import re
first_line_re = re.compile('^#!.*python[0-9.]*([ \t].*)?$')
class build_scripts(ori_build_scripts):

View File

@ -4,23 +4,24 @@ Created on 10 mars 2015
@author: coissac
'''
from sphinx.setup_command import BuildDoc as ori_build_sphinx # @UnresolvedImport
try:
from sphinx.setup_command import BuildDoc as ori_build_sphinx # @UnresolvedImport
class build_sphinx(ori_build_sphinx):
'''Build Sphinx documentation in html, epub and man formats
'''
class build_sphinx(ori_build_sphinx):
'''
Build Sphinx documentation in html, epub and man formats
'''
description = __doc__
def run(self):
self.builder='html'
self.finalize_options()
ori_build_sphinx.run(self)
self.builder='epub'
self.finalize_options()
ori_build_sphinx.run(self)
self.builder='man'
self.finalize_options()
ori_build_sphinx.run(self)
description = __doc__
def run(self):
self.builder='html'
self.finalize_options()
ori_build_sphinx.run(self)
self.builder='epub'
self.finalize_options()
ori_build_sphinx.run(self)
self.builder='man'
self.finalize_options()
ori_build_sphinx.run(self)
except ImportError:
pass

View File

@ -4,11 +4,16 @@ Created on 6 oct. 2014
@author: coissac
'''
# try:
# from setuptools.command.install import install as install_ori
# except ImportError:
# from distutils.command.install import install as install_ori
from distutils.command.install import install as install_ori
class install(install_ori):
def __init__(self,dist):
install_ori.__init__(self, dist)
self.sub_commands.insert(0, ('build',lambda self: True))
# self.sub_commands.insert(0, ('build',lambda self: True))
self.sub_commands.append(('install_sphinx',lambda self: self.distribution.serenity))

View File

@ -3,6 +3,12 @@ Created on 20 oct. 2012
@author: coissac
'''
# try:
# from setuptools.command.install_scripts import install_scripts as ori_install_scripts
# except ImportError:
# from distutils.command.install_scripts import install_scripts as ori_install_scripts
from distutils.command.install_scripts import install_scripts as ori_install_scripts
import os.path
@ -18,12 +24,10 @@ class install_scripts(ori_install_scripts):
def install_public_link(self):
self.mkpath(self.public_dir)
for file in self.get_outputs():
if self.dry_run:
log.info("changing mode of %s", file)
else:
log.info("exporting file %s -> %s", file,os.path.join(self.public_dir,
os.path.split(file)[1]
))
log.info("exporting file %s -> %s", file,os.path.join(self.public_dir,
os.path.split(file)[1]
))
if not self.dry_run:
dest = os.path.join(self.public_dir,
os.path.split(file)[1]
)

View File

@ -43,4 +43,19 @@ class install_sphinx(Command):
self.copy_file(os.path.join(epub),
os.path.join(self.install_doc,os.path.split(epub)[1]))
def get_outputs(self):
directory=os.path.join(self.install_doc,'html')
files = [os.path.join(self.install_doc,'html', f)
for dp, dn, filenames in os.walk(directory) for f in filenames] # @UnusedVariable
directory=os.path.join(self.build_dir,'man')
files.append(os.path.join(self.install_doc,'man','man1', f)
for dp, dn, filenames in os.walk(directory) for f in filenames) # @UnusedVariable
directory=os.path.join(self.build_dir,'epub')
files.append(os.path.join(self.install_doc, f)
for dp, dn, filenames in os.walk(directory) # @UnusedVariable
for f in glob.glob(os.path.join(dp, '*.epub')) )
return files

View File

@ -49,7 +49,7 @@ class littlebigman(build_exe):
shell=True,
stdout=subprocess.PIPE)
little = p.communicate()[0]
return little
return little.decode('latin1')
def run(self):
build_exe.run(self)

View File

@ -41,6 +41,10 @@ class pidname(build_exe):
else:
self.executables = []
# self.build_cexe = os.path.join(os.path.dirname(self.build_cexe),'cbinaries')
# self.mkpath(self.build_cexe)
def run(self):
if is_mac_system():

View File

@ -9,9 +9,12 @@ import os.path
import glob
import sys
try:
from setuptools.extension import Extension
except ImportError:
from distutils.extension import Extension
from distutils.core import setup as ori_setup
from distutils.extension import Extension
# from distutils.extension import Extension
from obidistutils.serenity.checkpackage import install_requirements,\
check_requirements, \
@ -22,6 +25,7 @@ from obidistutils.serenity.rerun import rerun_with_anothe_python
from distutils import log
from obidistutils.dist import Distribution
from obidistutils.serenity import is_serenity
def findPackage(root,base=None):
@ -42,11 +46,17 @@ def findCython(root,base=None,pyrexs=None):
base=[]
for module in (path.basename(path.dirname(x))
for x in glob.glob(path.join(root,'*','__init__.py'))):
for pyrex in glob.glob(path.join(root,module,'*.pyx')):
libabspath = os.path.abspath('obi_libdir')
obiabspath = os.path.abspath('.')
pyrexs.append(Extension('.'.join(base+[module,path.splitext(path.basename(pyrex))[0]]),
[pyrex]
[pyrex],
library_dirs=[libabspath],
include_dirs=[libabspath],
libraries=["obi3"],
runtime_library_dirs=[libabspath],
extra_link_args=["-Wl,-rpath,"+libabspath, "-L"+libabspath]
)
)
try:
@ -59,11 +69,16 @@ def findCython(root,base=None,pyrexs=None):
log.info("Cython module : %s",cfiles)
incdir = set(os.path.dirname(x) for x in cfiles if x[-2:]==".h")
cfiles = [x for x in cfiles if x[-2:]==".c"]
pyrexs[-1].sources.extend(cfiles)
#cfiles = [x for x in cfiles if x[-2:]==".c"]
#pyrexs[-1].sources.extend(cfiles)
pyrexs[-1].include_dirs.extend(incdir)
pyrexs[-1].extra_compile_args.extend(['-msse2','-Wno-unused-function'])
pyrexs[-1].extra_compile_args.extend(['-msse2',
'-Wno-unused-function',
'-Wmissing-braces',
'-Wchar-subscripts',
'-fPIC'
])
except IOError:
pass
@ -78,8 +93,8 @@ def rootname(x):
def prepare_commands():
from obidistutils.command.build import build
from obidistutils.command.littlebigman import littlebigman
# from obidistutils.command.serenity import serenity
from obidistutils.command.build_cexe import build_cexe
from obidistutils.command.build_sphinx import build_sphinx
from obidistutils.command.build_ext import build_ext
from obidistutils.command.build_ctools import build_ctools
from obidistutils.command.build_files import build_files
@ -89,8 +104,11 @@ def prepare_commands():
from obidistutils.command.install import install
from obidistutils.command.pidname import pidname
from obidistutils.command.sdist import sdist
COMMANDS = {'build':build,
# 'serenity':serenity,
'littlebigman':littlebigman,
'pidname':pidname,
'build_ctools':build_ctools,
@ -98,12 +116,22 @@ def prepare_commands():
'build_cexe':build_cexe,
'build_ext': build_ext,
'build_scripts':build_scripts,
'build_sphinx':build_sphinx,
'install_scripts':install_scripts,
'install_sphinx':install_sphinx,
'install':install,
'sdist':sdist}
# try:
# from setuptools.commands import egg_info
# COMMANDS['egg_info']=egg_info
# except ImportError:
# pass
try:
from obidistutils.command.build_sphinx import build_sphinx
COMMANDS['build_sphinx']=build_sphinx
except ImportError:
pass
return COMMANDS
@ -115,7 +143,7 @@ def setup(**attrs):
log.set_threshold(log.INFO)
minversion = attrs.get("pythonmin",'3.4')
minversion = attrs.get("pythonmin",'3.7')
maxversion = attrs.get('pythonmax',None)
fork = attrs.get('fork',False)
requirementfile = attrs.get('requirements','requirements.txt')
@ -139,19 +167,21 @@ def setup(**attrs):
del attrs['requirements']
except KeyError:
pass
enforce_good_python(minversion, maxversion, fork)
if is_serenity():
if (install_requirements(requirementfile)):
rerun_with_anothe_python(sys.executable,minversion,maxversion,fork)
enforce_good_python(minversion, maxversion, fork)
if (install_requirements(requirementfile)):
rerun_with_anothe_python(sys.executable,minversion,maxversion,fork)
try:
check_requirements(requirementfile)
except RequirementError as e :
log.error(e)
sys.exit(1)
try:
check_requirements(requirementfile)
except RequirementError as e :
log.error(e)
sys.exit(1)
if 'distclass' not in attrs:
attrs['distclass']=Distribution
@ -193,5 +223,12 @@ def setup(**attrs):
if 'ext_modules' not in attrs:
attrs['ext_modules'] = EXTENTION
# try:
# from setuptools.core import setup as ori_setup
# except ImportError:
# from distutils.core import setup as ori_setup
from distutils.core import setup as ori_setup
ori_setup(**attrs)
return ori_setup(**attrs)

View File

@ -4,7 +4,12 @@ Created on 20 oct. 2012
@author: coissac
'''
from distutils.dist import Distribution as ori_Distribution
try:
from setuptools.dist import Distribution as ori_Distribution
except ImportError:
from distutils.dist import Distribution as ori_Distribution
# from distutils.dist import Distribution as ori_Distribution
class Distribution(ori_Distribution):
@ -29,6 +34,14 @@ class Distribution(ori_Distribution):
"By default the name is PACKAGE-VERSION"
))
def run_commands(self):
"""Run each command that was seen on the setup script command line.
Uses the list of commands found and cache of command objects
created by 'get_command_obj()'.
"""
# self.run_command('littlebigman')
ori_Distribution.run_commands(self)
def has_executables(self):
return self.executables is not None and self.executables

View File

@ -11,8 +11,7 @@ import re
from distutils.errors import DistutilsError
import tempfile
import importlib
import imp
from importlib.util import spec_from_file_location # @UnresolvedImport
import zipimport
import argparse
@ -82,9 +81,15 @@ def serenity_mode(package,version):
argparser.add_argument('--serenity',
dest='serenity',
action='store_true',
default=False,
default=True,
help='Switch the installer in serenity mode. Everythings are installed in a virtualenv')
argparser.add_argument('--no-serenity',
dest='serenity',
action='store_false',
default=True,
help='Switch the installer in the no serenity mode.')
argparser.add_argument('--virtualenv',
dest='virtual',
type=str,
@ -106,3 +111,8 @@ def serenity_mode(package,version):
return args.serenity
def getVersion(source,main,version):
path = os.path.join(source,main,'%s.py' % version)
spec = spec_from_file_location('version',path)
return spec.loader.load_module().version.strip()

View File

@ -0,0 +1,36 @@
'''
Created on 22 janv. 2016
@author: coissac
'''
import sys
from urllib import request
import os.path
from obidistutils.serenity.util import get_serenity_dir
from obidistutils.serenity.rerun import rerun_with_anothe_python
from obidistutils.serenity.checkpython import is_a_virtualenv_python
getpipurl="https://bootstrap.pypa.io/get-pip.py"
def bootstrap():
getpipfile=os.path.join(get_serenity_dir(),"get-pip.py")
with request.urlopen(getpipurl) as getpip:
with open(getpipfile,"wb") as out:
for l in getpip:
out.write(l)
python = sys.executable
if is_a_virtualenv_python():
command= "%s %s" % (python,getpipfile)
else:
command= "%s %s --user" % (python,getpipfile)
os.system(command)
rerun_with_anothe_python(python)

View File

@ -5,27 +5,35 @@ Created on 2 oct. 2014
'''
import re
import os
import pip # @UnresolvedImport
from pip.utils import get_installed_distributions # @UnresolvedImport
from distutils.version import StrictVersion # @UnusedImport
from distutils.errors import DistutilsError
from distutils import log
import os.path
import sys
import subprocess
class RequirementError(Exception):
pass
def is_installed(requirement):
pipcommand = os.path.join(os.path.dirname(sys.executable),'pip')
pipjson = subprocess.run([pipcommand,"list","--format=json"],
capture_output=True).stdout
packages = eval(pipjson)
requirement_project,requirement_relation,requirement_version = parse_package_requirement(requirement)
package = [x for x in get_installed_distributions() if x.project_name==requirement_project]
package = [x for x in packages if x["name"]==requirement_project]
if len(package)==1:
if requirement_version is not None and requirement_relation is not None:
rep = (len(package)==1) and eval("StrictVersion('%s') %s StrictVersion('%s')" % (package[0].version,
if ( requirement_version is not None
and requirement_relation is not None):
rep = (len(package)==1) and eval("StrictVersion('%s') %s StrictVersion('%s')" % (package[0]["version"],
requirement_relation,
requirement_version)
)
@ -39,20 +47,23 @@ def is_installed(requirement):
log.info("Look for package %s (%s%s) : ok version %s installed" % (requirement_project,
requirement_relation,
requirement_version,
package[0].version))
package[0]["version"]))
else:
log.info("Look for package %s : ok version %s installed" % (requirement_project,
package[0].version))
package[0]["version"]))
else:
if len(package)!=1:
log.info("Look for package %s (%s%s) : not installed" % (requirement_project,
requirement_relation,
requirement_version))
if requirement_version is not None and requirement_relation is not None:
log.info("Look for package %s (%s%s) : not installed" % (requirement_project,
requirement_relation,
requirement_version))
else:
log.info("Look for package %s : not installed" % requirement_project)
else:
log.info("Look for package %s (%s%s) : failed only version %s installed" % (requirement_project,
requirement_relation,
requirement_version,
package[0].version))
package[0]["version"]))
return rep
@ -81,8 +92,10 @@ def install_requirements(requirementfile='requirements.txt'):
ok = is_installed(x)
if not ok:
log.info(" Installing requirement : %s" % x)
pip_install_package(x)
pip_install_package(x,requirement=requirementfile)
install_something=True
if x[0:3]=='pip':
return True
return install_something
@ -132,25 +145,31 @@ def get_package_requirement(package,requirementfile='requirements.txt'):
return None
def pip_install_package(package,directory=None,upgrade=True):
def pip_install_package(package,directory=None,requirement=None):
log.info('installing %s in directory %s' % (package,str(directory)))
pipcommand = os.path.join(os.path.dirname(sys.executable),'pip')
if directory is not None:
log.info(' installing %s in directory %s' % (package,str(directory)))
if 'http_proxy' in os.environ and 'https_proxy' not in os.environ:
os.environ['https_proxy']=os.environ['http_proxy']
args = ['install']
if upgrade:
args.append('--upgrade')
if requirement:
args.append('--requirement')
args.append(requirement)
if 'http_proxy' in os.environ:
args.append('--proxy=%s' % os.environ['http_proxy'])
if 'https_proxy' in os.environ:
args.append('--proxy=%s' % os.environ['https_proxy'])
if directory is not None:
args.append('--target=%s' % directory)
args.append(package)
return pip.main(args)
pip = subprocess.run([pipcommand] + args)
return pip

View File

@ -37,7 +37,7 @@ def is_python_version(path=None,minversion='3.4',maxversion=None):
stdout=subprocess.PIPE)
pythonversion=str(p.communicate()[0],'utf8').strip()
pythonversion = StrictVersion(pythonversion)
return ( pythonversion >=StrictVersion(minversion)
and ( maxversion is None
or pythonversion < StrictVersion(maxversion))
@ -91,9 +91,9 @@ def is_a_virtualenv_python(path=None):
'''
if path is None:
rep = sys.base_exec_prefix == sys.exec_prefix
rep = sys.base_exec_prefix != sys.exec_prefix
else:
command = """'%s' -c 'import sys; print(sys.base_exec_prefix == sys.exec_prefix)'""" % path
command = """'%s' -c 'import sys; print(sys.base_exec_prefix != sys.exec_prefix)'""" % path
p = subprocess.Popen(command,
shell=True,
stdout=subprocess.PIPE)

View File

@ -9,10 +9,10 @@ import sys
import venv
from distutils.errors import DistutilsError
from obidistutils.serenity.globals import local_virtualenv # @UnusedImport
from obidistutils.serenity.checkpython import which_virtualenv,\
is_python_version, \
is_a_virtualenv_python
from .globals import local_virtualenv # @UnusedImport
from .checkpython import which_virtualenv,\
is_python_version, \
is_a_virtualenv_python
@ -39,6 +39,7 @@ def serenity_virtualenv(envname,package,version,minversion='3.4',maxversion=None
maxversion=maxversion) and
is_a_virtualenv_python(python))
#
# The virtualenv already exist but it is not ok
#
@ -58,7 +59,7 @@ def serenity_virtualenv(envname,package,version,minversion='3.4',maxversion=None
clear=True,
symlinks=False,
with_pip=True)
# check the newly created virtualenv
return serenity_virtualenv(envname,package,version)

View File

@ -5,7 +5,7 @@
* Author: coissac
*/
#include<stdio.h>
#include <stdio.h>
int main(int argc, char *argv[])
{

2
doc/.gitignore vendored Normal file → Executable file
View File

@ -1,3 +1,5 @@
/build/
/doxygen/
/build_dir.txt
/.DS_Store
/.gitignore

0
doc/Doxyfile Normal file → Executable file
View File

2
doc/Makefile Normal file → Executable file
View File

@ -57,7 +57,7 @@ html:
@echo "Generating Doxygen documentation..."
doxygen Doxyfile
@echo "Doxygen documentation generated. \n"
$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
$(SPHINXBUILD) -b html -c ./ $(ALLSPHINXOPTS) $(BUILDDIR)/html
@echo
@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."

View File

@ -1,16 +0,0 @@
__version__ = '4.0.0'
def setup(app):
# We can't do the import at the module scope as setup.py has to be able to
# import this file to read __version__ without hitting any syntax errors
# from both Python 2 & Python 3.
# By the time this function is called, the directives code will have been
# converted with 2to3 if appropriate
from . import directives
directives.setup(app)

Binary file not shown.

Binary file not shown.

View File

@ -1,88 +0,0 @@
from ..renderer.rst.doxygen.base import RenderContext
from ..renderer.rst.doxygen import format_parser_error
from ..parser import ParserError, FileIOError
from docutils import nodes
from docutils.parsers import rst
class WarningHandler(object):
def __init__(self, state, context):
self.state = state
self.context = context
def warn(self, raw_text, rendered_nodes=None):
raw_text = self.format(raw_text)
if rendered_nodes is None:
rendered_nodes = [nodes.paragraph("", "", nodes.Text(raw_text))]
return [
nodes.warning("", *rendered_nodes),
self.state.document.reporter.warning(raw_text, line=self.context['lineno'])
]
def format(self, text):
return text.format(**self.context)
def create_warning(project_info, state, lineno, **kwargs):
tail = ''
if project_info:
tail = 'in doxygen xml output for project "{project}" from directory: {path}'.format(
project=project_info.name(),
path=project_info.project_path()
)
context = dict(
lineno=lineno,
tail=tail,
**kwargs
)
return WarningHandler(state, context)
class BaseDirective(rst.Directive):
def __init__(self, root_data_object, renderer_factory_creator_constructor, finder_factory,
project_info_factory, filter_factory, target_handler_factory, parser_factory, *args):
rst.Directive.__init__(self, *args)
self.directive_args = list(args) # Convert tuple to list to allow modification.
self.root_data_object = root_data_object
self.renderer_factory_creator_constructor = renderer_factory_creator_constructor
self.finder_factory = finder_factory
self.project_info_factory = project_info_factory
self.filter_factory = filter_factory
self.target_handler_factory = target_handler_factory
self.parser_factory = parser_factory
def render(self, node_stack, project_info, options, filter_, target_handler, mask_factory):
"Standard render process used by subclasses"
renderer_factory_creator = self.renderer_factory_creator_constructor.create_factory_creator(
project_info,
self.state.document,
options,
target_handler
)
try:
renderer_factory = renderer_factory_creator.create_factory(
node_stack,
self.state,
self.state.document,
filter_,
target_handler,
)
except ParserError as e:
return format_parser_error("doxygenclass", e.error, e.filename, self.state,
self.lineno, True)
except FileIOError as e:
return format_parser_error("doxygenclass", e.error, e.filename, self.state, self.lineno)
context = RenderContext(node_stack, mask_factory, self.directive_args)
object_renderer = renderer_factory.create_renderer(context)
return object_renderer.render()

Binary file not shown.

View File

@ -1,123 +0,0 @@
from ..renderer.rst.doxygen.base import RenderContext
from ..renderer.rst.doxygen.mask import NullMaskFactory
from ..directive.base import BaseDirective
from ..project import ProjectError
from .base import create_warning
from docutils.parsers.rst.directives import unchanged_required, flag
from docutils.parsers import rst
class BaseFileDirective(BaseDirective):
"""Base class handle the main work when given the appropriate file and project info to work
from.
"""
# We use inheritance here rather than a separate object and composition, because so much
# information is present in the Directive class from the docutils framework that we'd have to
# pass way too much stuff to a helper object to be reasonable.
def handle_contents(self, file_, project_info):
finder = self.finder_factory.create_finder(project_info)
finder_filter = self.filter_factory.create_file_finder_filter(file_)
matches = []
finder.filter_(finder_filter, matches)
if len(matches) > 1:
warning = create_warning(None, self.state, self.lineno, file=file_,
directivename=self.directive_name)
return warning.warn('{directivename}: Found multiple matches for file "{file} {tail}')
elif not matches:
warning = create_warning(None, self.state, self.lineno, file=file_,
directivename=self.directive_name)
return warning.warn('{directivename}: Cannot find file "{file} {tail}')
target_handler = self.target_handler_factory.create_target_handler(
self.options, project_info, self.state.document)
filter_ = self.filter_factory.create_file_filter(file_, self.options)
renderer_factory_creator = self.renderer_factory_creator_constructor.create_factory_creator(
project_info,
self.state.document,
self.options,
target_handler
)
node_list = []
for node_stack in matches:
renderer_factory = renderer_factory_creator.create_factory(
node_stack,
self.state,
self.state.document,
filter_,
target_handler,
)
mask_factory = NullMaskFactory()
context = RenderContext(node_stack, mask_factory, self.directive_args)
object_renderer = renderer_factory.create_renderer(context)
node_list.extend(object_renderer.render())
return node_list
class DoxygenFileDirective(BaseFileDirective):
directive_name = 'doxygenfile'
required_arguments = 0
optional_arguments = 2
option_spec = {
"path": unchanged_required,
"project": unchanged_required,
"outline": flag,
"no-link": flag,
}
has_content = False
def run(self):
"""Get the file from the argument and the project info from the factory."""
file_ = self.arguments[0]
try:
project_info = self.project_info_factory.create_project_info(self.options)
except ProjectError as e:
warning = create_warning(None, self.state, self.lineno)
return warning.warn('doxygenfile: %s' % e)
return self.handle_contents(file_, project_info)
class AutoDoxygenFileDirective(BaseFileDirective):
directive_name = 'autodoxygenfile'
required_arguments = 1
option_spec = {
"project": unchanged_required,
"outline": flag,
"no-link": flag,
}
has_content = False
def run(self):
"""Get the file from the argument and extract the associated project info for the named
project given that it is an auto-project.
"""
file_ = self.arguments[0]
try:
project_info = self.project_info_factory.retrieve_project_info_for_auto(self.options)
except ProjectError as e:
warning = create_warning(None, self.state, self.lineno)
return warning.warn('autodoxygenfile: %s' % e)
return self.handle_contents(file_, project_info)

Binary file not shown.

View File

@ -1,115 +0,0 @@
from ..renderer.rst.doxygen.base import RenderContext
from ..renderer.rst.doxygen.mask import NullMaskFactory
from ..renderer.rst.doxygen import format_parser_error
from ..directive.base import BaseDirective
from ..project import ProjectError
from ..parser import ParserError, FileIOError
from .base import create_warning
from docutils.parsers import rst
from docutils.parsers.rst.directives import unchanged_required, flag
class BaseIndexDirective(BaseDirective):
"""Base class handle the main work when given the appropriate project info to work from.
"""
# We use inheritance here rather than a separate object and composition, because so much
# information is present in the Directive class from the docutils framework that we'd have to
# pass way too much stuff to a helper object to be reasonable.
def handle_contents(self, project_info):
try:
finder = self.finder_factory.create_finder(project_info)
except ParserError as e:
return format_parser_error(self.name, e.error, e.filename, self.state,
self.lineno, True)
except FileIOError as e:
return format_parser_error(self.name, e.error, e.filename, self.state, self.lineno)
data_object = finder.root()
target_handler = self.target_handler_factory.create_target_handler(
self.options, project_info, self.state.document)
filter_ = self.filter_factory.create_index_filter(self.options)
renderer_factory_creator = self.renderer_factory_creator_constructor.create_factory_creator(
project_info,
self.state.document,
self.options,
target_handler
)
renderer_factory = renderer_factory_creator.create_factory(
[data_object],
self.state,
self.state.document,
filter_,
target_handler,
)
mask_factory = NullMaskFactory()
context = RenderContext([data_object, self.root_data_object], mask_factory, self.directive_args)
object_renderer = renderer_factory.create_renderer(context)
try:
node_list = object_renderer.render()
except ParserError as e:
return format_parser_error(self.name, e.error, e.filename, self.state,
self.lineno, True)
except FileIOError as e:
return format_parser_error(self.name, e.error, e.filename, self.state, self.lineno)
return node_list
class DoxygenIndexDirective(BaseIndexDirective):
required_arguments = 0
optional_arguments = 2
option_spec = {
"path": unchanged_required,
"project": unchanged_required,
"outline": flag,
"no-link": flag,
}
has_content = False
def run(self):
"""Extract the project info and pass it to the helper method"""
try:
project_info = self.project_info_factory.create_project_info(self.options)
except ProjectError as e:
warning = create_warning(None, self.state, self.lineno)
return warning.warn('doxygenindex: %s' % e)
return self.handle_contents(project_info)
class AutoDoxygenIndexDirective(BaseIndexDirective):
required_arguments = 0
final_argument_whitespace = True
option_spec = {
"project": unchanged_required,
"outline": flag,
"no-link": flag,
}
has_content = False
def run(self):
"""Extract the project info from the auto project info store and pass it to the helper
method
"""
try:
project_info = self.project_info_factory.retrieve_project_info_for_auto(self.options)
except ProjectError as e:
warning = create_warning(None, self.state, self.lineno)
return warning.warn('autodoxygenindex: %s' % e)
return self.handle_contents(project_info)

Binary file not shown.

File diff suppressed because it is too large Load Diff

Binary file not shown.

View File

@ -1,3 +0,0 @@
class BreatheError(Exception):
pass

Binary file not shown.

Binary file not shown.

View File

@ -1,45 +0,0 @@
class FakeParentNode(object):
node_type = "fakeparent"
class Finder(object):
def __init__(self, root, item_finder_factory):
self._root = root
self.item_finder_factory = item_finder_factory
def filter_(self, filter_, matches):
"""Adds all nodes which match the filter into the matches list"""
item_finder = self.item_finder_factory.create_finder(self._root)
item_finder.filter_([FakeParentNode()], filter_, matches)
def root(self):
return self._root
class FinderFactory(object):
def __init__(self, parser, item_finder_factory_creator):
self.parser = parser
self.item_finder_factory_creator = item_finder_factory_creator
def create_finder(self, project_info):
root = self.parser.parse(project_info)
item_finder_factory = self.item_finder_factory_creator.create_factory(project_info)
return Finder(root, item_finder_factory)
def create_finder_from_root(self, root, project_info):
item_finder_factory = self.item_finder_factory_creator.create_factory(project_info)
return Finder(root, item_finder_factory)

Binary file not shown.

View File

@ -1 +0,0 @@

View File

@ -1,18 +0,0 @@
class ItemFinder(object):
def __init__(self, project_info, data_object, item_finder_factory):
self.data_object = data_object
self.item_finder_factory = item_finder_factory
self.project_info = project_info
def stack(element, list_):
"""Stack an element on to the start of a list and return as a new list"""
# Copy list first so we have a new list to insert into
output = list_[:]
output.insert(0, element)
return output

Binary file not shown.

View File

@ -1,75 +0,0 @@
from .base import ItemFinder, stack
class DoxygenTypeSubItemFinder(ItemFinder):
def filter_(self, ancestors, filter_, matches):
"""Find nodes which match the filter. Doesn't test this node, only its children"""
node_stack = stack(self.data_object, ancestors)
compound_finder = self.item_finder_factory.create_finder(self.data_object.compounddef)
compound_finder.filter_(node_stack, filter_, matches)
class CompoundDefTypeSubItemFinder(ItemFinder):
def filter_(self, ancestors, filter_, matches):
"""Finds nodes which match the filter and continues checks to children"""
node_stack = stack(self.data_object, ancestors)
if filter_.allow(node_stack):
matches.append(node_stack)
for sectiondef in self.data_object.sectiondef:
finder = self.item_finder_factory.create_finder(sectiondef)
finder.filter_(node_stack, filter_, matches)
for innerclass in self.data_object.innerclass:
finder = self.item_finder_factory.create_finder(innerclass)
finder.filter_(node_stack, filter_, matches)
class SectionDefTypeSubItemFinder(ItemFinder):
def filter_(self, ancestors, filter_, matches):
"""Find nodes which match the filter. Doesn't test this node, only its children"""
node_stack = stack(self.data_object, ancestors)
if filter_.allow(node_stack):
matches.append(node_stack)
for memberdef in self.data_object.memberdef:
finder = self.item_finder_factory.create_finder(memberdef)
finder.filter_(node_stack, filter_, matches)
class MemberDefTypeSubItemFinder(ItemFinder):
def filter_(self, ancestors, filter_, matches):
data_object = self.data_object
node_stack = stack(data_object, ancestors)
if filter_.allow(node_stack):
matches.append(node_stack)
if data_object.kind == 'enum':
for value in data_object.enumvalue:
value_stack = stack(value, node_stack)
if filter_.allow(value_stack):
matches.append(value_stack)
class RefTypeSubItemFinder(ItemFinder):
def filter_(self, ancestors, filter_, matches):
node_stack = stack(self.data_object, ancestors)
if filter_.allow(node_stack):
matches.append(node_stack)

View File

@ -1,55 +0,0 @@
from . import index as indexfinder
from . import compound as compoundfinder
class CreateCompoundTypeSubFinder(object):
def __init__(self, parser_factory, matcher_factory):
self.parser_factory = parser_factory
self.matcher_factory = matcher_factory
def __call__(self, project_info, *args):
compound_parser = self.parser_factory.create_compound_parser(project_info)
return indexfinder.CompoundTypeSubItemFinder(self.matcher_factory, compound_parser,
project_info, *args)
class DoxygenItemFinderFactory(object):
def __init__(self, finders, project_info):
self.finders = finders
self.project_info = project_info
def create_finder(self, data_object):
return self.finders[data_object.node_type](self.project_info, data_object, self)
class DoxygenItemFinderFactoryCreator(object):
def __init__(self, parser_factory, filter_factory):
self.parser_factory = parser_factory
self.filter_factory = filter_factory
def create_factory(self, project_info):
finders = {
"doxygen": indexfinder.DoxygenTypeSubItemFinder,
"compound": CreateCompoundTypeSubFinder(self.parser_factory, self.filter_factory),
"member": indexfinder.MemberTypeSubItemFinder,
"doxygendef": compoundfinder.DoxygenTypeSubItemFinder,
"compounddef": compoundfinder.CompoundDefTypeSubItemFinder,
"sectiondef": compoundfinder.SectionDefTypeSubItemFinder,
"memberdef": compoundfinder.MemberDefTypeSubItemFinder,
"ref": compoundfinder.RefTypeSubItemFinder,
}
return DoxygenItemFinderFactory(finders, project_info)

Binary file not shown.

View File

@ -1,79 +0,0 @@
from .base import ItemFinder, stack
class DoxygenTypeSubItemFinder(ItemFinder):
def filter_(self, ancestors, filter_, matches):
"""Find nodes which match the filter. Doesn't test this node, only its children"""
compounds = self.data_object.get_compound()
node_stack = stack(self.data_object, ancestors)
for compound in compounds:
compound_finder = self.item_finder_factory.create_finder(compound)
compound_finder.filter_(node_stack, filter_, matches)
class CompoundTypeSubItemFinder(ItemFinder):
def __init__(self, filter_factory, compound_parser, *args):
ItemFinder.__init__(self, *args)
self.filter_factory = filter_factory
self.compound_parser = compound_parser
def filter_(self, ancestors, filter_, matches):
"""Finds nodes which match the filter and continues checks to children
Requires parsing the xml files referenced by the children for which we use the compound
parser and continue at the top level of that pretending that this node is the parent of the
top level node of the compound file.
"""
node_stack = stack(self.data_object, ancestors)
# Match against compound object
if filter_.allow(node_stack):
matches.append(node_stack)
# Descend to member children
members = self.data_object.get_member()
member_matches = []
for member in members:
member_finder = self.item_finder_factory.create_finder(member)
member_finder.filter_(node_stack, filter_, member_matches)
results = []
# If there are members in this compound that match the criteria
# then load up the file for this compound and get the member data objects
if member_matches:
file_data = self.compound_parser.parse(self.data_object.refid)
finder = self.item_finder_factory.create_finder(file_data)
for member_stack in member_matches:
ref_filter = self.filter_factory.create_id_filter('memberdef', member_stack[0].refid)
finder.filter_(node_stack, ref_filter, matches)
else:
# Read in the xml file referenced by the compound and descend into that as well
file_data = self.compound_parser.parse(self.data_object.refid)
finder = self.item_finder_factory.create_finder(file_data)
finder.filter_(node_stack, filter_, matches)
class MemberTypeSubItemFinder(ItemFinder):
def filter_(self, ancestors, filter_, matches):
node_stack = stack(self.data_object, ancestors)
# Match against member object
if filter_.allow(node_stack):
matches.append(node_stack)

Binary file not shown.

View File

@ -1,118 +0,0 @@
import breathe.parser.doxygen.index
import breathe.parser.doxygen.compound
class ParserError(Exception):
def __init__(self, error, filename):
Exception.__init__(self, error)
self.error = error
self.filename = filename
class FileIOError(Exception):
def __init__(self, error, filename):
Exception.__init__(self, error)
self.error = error
self.filename = filename
class Parser(object):
def __init__(self, cache, path_handler, file_state_cache):
self.cache = cache
self.path_handler = path_handler
self.file_state_cache = file_state_cache
class DoxygenIndexParser(Parser):
def __init__(self, cache, path_handler, file_state_cache):
Parser.__init__(self, cache, path_handler, file_state_cache)
def parse(self, project_info):
filename = self.path_handler.resolve_path(
project_info.project_path(),
"index.xml"
)
self.file_state_cache.update(filename)
try:
# Try to get from our cache
return self.cache[filename]
except KeyError:
# If that fails, parse it afresh
try:
result = breathe.parser.doxygen.index.parse(filename)
self.cache[filename] = result
return result
except breathe.parser.doxygen.index.ParseError as e:
raise ParserError(e, filename)
except breathe.parser.doxygen.index.FileIOError as e:
raise FileIOError(e, filename)
class DoxygenCompoundParser(Parser):
def __init__(self, cache, path_handler, file_state_cache, project_info):
Parser.__init__(self, cache, path_handler, file_state_cache)
self.project_info = project_info
def parse(self, refid):
filename = self.path_handler.resolve_path(
self.project_info.project_path(),
"%s.xml" % refid
)
self.file_state_cache.update(filename)
try:
# Try to get from our cache
return self.cache[filename]
except KeyError:
# If that fails, parse it afresh
try:
result = breathe.parser.doxygen.compound.parse(filename)
self.cache[filename] = result
return result
except breathe.parser.doxygen.compound.ParseError as e:
raise ParserError(e, filename)
except breathe.parser.doxygen.compound.FileIOError as e:
raise FileIOError(e, filename)
class CacheFactory(object):
def create_cache(self):
# Return basic dictionary as cache
return {}
class DoxygenParserFactory(object):
def __init__(self, cache, path_handler, file_state_cache):
self.cache = cache
self.path_handler = path_handler
self.file_state_cache = file_state_cache
def create_index_parser(self):
return DoxygenIndexParser(self.cache, self.path_handler, self.file_state_cache)
def create_compound_parser(self, project_info):
return DoxygenCompoundParser(
self.cache,
self.path_handler,
self.file_state_cache,
project_info
)

Binary file not shown.

View File

@ -1,964 +0,0 @@
#!/usr/bin/env python
"""
Generated Mon Feb 9 19:08:05 2009 by generateDS.py.
"""
from xml.dom import minidom
from xml.dom import Node
from xml.parsers.expat import ExpatError
from . import compoundsuper as supermod
from .compoundsuper import MixedContainer
class DoxygenTypeSub(supermod.DoxygenType):
node_type = "doxygendef"
def __init__(self, version=None, compounddef=None):
supermod.DoxygenType.__init__(self, version, compounddef)
supermod.DoxygenType.subclass = DoxygenTypeSub
# end class DoxygenTypeSub
class compounddefTypeSub(supermod.compounddefType):
node_type = "compounddef"
def __init__(self, kind=None, prot=None, id=None, compoundname='', title='',
basecompoundref=None, derivedcompoundref=None, includes=None, includedby=None,
incdepgraph=None, invincdepgraph=None, innerdir=None, innerfile=None,
innerclass=None, innernamespace=None, innerpage=None, innergroup=None,
templateparamlist=None, sectiondef=None, briefdescription=None,
detaileddescription=None, inheritancegraph=None, collaborationgraph=None,
programlisting=None, location=None, listofallmembers=None):
supermod.compounddefType.__init__(self, kind, prot, id, compoundname, title,
basecompoundref, derivedcompoundref, includes, includedby,
incdepgraph, invincdepgraph, innerdir, innerfile,
innerclass, innernamespace, innerpage, innergroup,
templateparamlist, sectiondef, briefdescription,
detaileddescription, inheritancegraph, collaborationgraph,
programlisting, location, listofallmembers)
supermod.compounddefType.subclass = compounddefTypeSub
# end class compounddefTypeSub
class listofallmembersTypeSub(supermod.listofallmembersType):
node_type = "listofallmembers"
def __init__(self, member=None):
supermod.listofallmembersType.__init__(self, member)
supermod.listofallmembersType.subclass = listofallmembersTypeSub
# end class listofallmembersTypeSub
class memberRefTypeSub(supermod.memberRefType):
node_type = "memberref"
def __init__(self, virt=None, prot=None, refid=None, ambiguityscope=None, scope='', name=''):
supermod.memberRefType.__init__(self, virt, prot, refid, ambiguityscope, scope, name)
supermod.memberRefType.subclass = memberRefTypeSub
# end class memberRefTypeSub
class compoundRefTypeSub(supermod.compoundRefType):
node_type = "compoundref"
def __init__(self, virt=None, prot=None, refid=None, valueOf_='', mixedclass_=None,
content_=None):
supermod.compoundRefType.__init__(self, mixedclass_, content_)
supermod.compoundRefType.subclass = compoundRefTypeSub
# end class compoundRefTypeSub
class reimplementTypeSub(supermod.reimplementType):
node_type = "reimplement"
def __init__(self, refid=None, valueOf_='', mixedclass_=None, content_=None):
supermod.reimplementType.__init__(self, mixedclass_, content_)
supermod.reimplementType.subclass = reimplementTypeSub
# end class reimplementTypeSub
class incTypeSub(supermod.incType):
node_type = "inc"
def __init__(self, local=None, refid=None, valueOf_='', mixedclass_=None, content_=None):
supermod.incType.__init__(self, mixedclass_, content_)
supermod.incType.subclass = incTypeSub
# end class incTypeSub
class refTypeSub(supermod.refType):
node_type = "ref"
def __init__(self, node_name, prot=None, refid=None, valueOf_='', mixedclass_=None,
content_=None):
supermod.refType.__init__(self, mixedclass_, content_)
self.node_name = node_name
supermod.refType.subclass = refTypeSub
class refTextTypeSub(supermod.refTextType):
node_type = "reftex"
def __init__(self, refid=None, kindref=None, external=None, valueOf_='', mixedclass_=None,
content_=None):
supermod.refTextType.__init__(self, mixedclass_, content_)
supermod.refTextType.subclass = refTextTypeSub
# end class refTextTypeSub
class sectiondefTypeSub(supermod.sectiondefType):
node_type = "sectiondef"
def __init__(self, kind=None, header='', description=None, memberdef=None):
supermod.sectiondefType.__init__(self, kind, header, description, memberdef)
supermod.sectiondefType.subclass = sectiondefTypeSub
# end class sectiondefTypeSub
class memberdefTypeSub(supermod.memberdefType):
node_type = "memberdef"
def __init__(self, initonly=None, kind=None, volatile=None, const=None, raise_=None, virt=None,
readable=None, prot=None, explicit=None, new=None, final=None, writable=None,
add=None, static=None, remove=None, sealed=None, mutable=None, gettable=None,
inline=None, settable=None, id=None, templateparamlist=None, type_=None,
definition='', argsstring='', name='', read='', write='', bitfield='',
reimplements=None, reimplementedby=None, param=None, enumvalue=None,
initializer=None, exceptions=None, briefdescription=None, detaileddescription=None,
inbodydescription=None, location=None, references=None, referencedby=None):
supermod.memberdefType.__init__(self, initonly, kind, volatile, const, raise_, virt,
readable, prot, explicit, new, final, writable, add, static,
remove, sealed, mutable, gettable, inline, settable, id,
templateparamlist, type_, definition, argsstring, name,
read, write, bitfield, reimplements, reimplementedby, param,
enumvalue, initializer, exceptions, briefdescription,
detaileddescription, inbodydescription, location,
references, referencedby)
self.parameterlist = supermod.docParamListType.factory()
self.parameterlist.kind = "param"
def buildChildren(self, child_, nodeName_):
supermod.memberdefType.buildChildren(self, child_, nodeName_)
if child_.nodeType == Node.ELEMENT_NODE and nodeName_ == 'param':
# Get latest param
param = self.param[-1]
# If it doesn't have a description we're done
if not param.briefdescription:
return
# Construct our own param list from the descriptions stored inline
# with the parameters
paramdescription = param.briefdescription
paramname = supermod.docParamName.factory()
# Add parameter name
obj_ = paramname.mixedclass_(MixedContainer.CategoryText, MixedContainer.TypeNone, '',
param.declname)
paramname.content_.append(obj_)
paramnamelist = supermod.docParamNameList.factory()
paramnamelist.parametername.append(paramname)
paramlistitem = supermod.docParamListItem.factory()
paramlistitem.parameternamelist.append(paramnamelist)
# Add parameter description
paramlistitem.parameterdescription = paramdescription
self.parameterlist.parameteritem.append(paramlistitem)
elif child_.nodeType == Node.ELEMENT_NODE and nodeName_ == 'detaileddescription':
if not self.parameterlist.parameteritem:
# No items in our list
return
# Assume supermod.memberdefType.buildChildren has already built the
# description object, we just want to slot our parameterlist in at
# a reasonable point
if not self.detaileddescription:
# Create one if it doesn't exist
self.detaileddescription = supermod.descriptionType.factory()
detaileddescription = self.detaileddescription
para = supermod.docParaType.factory()
para.parameterlist.append(self.parameterlist)
obj_ = detaileddescription.mixedclass_(MixedContainer.CategoryComplex,
MixedContainer.TypeNone, 'para', para)
index = 0
detaileddescription.content_.insert(index, obj_)
supermod.memberdefType.subclass = memberdefTypeSub
# end class memberdefTypeSub
class descriptionTypeSub(supermod.descriptionType):
node_type = "description"
def __init__(self, title='', para=None, sect1=None, internal=None, mixedclass_=None,
content_=None):
supermod.descriptionType.__init__(self, mixedclass_, content_)
supermod.descriptionType.subclass = descriptionTypeSub
# end class descriptionTypeSub
class enumvalueTypeSub(supermod.enumvalueType):
node_type = "enumvalue"
def __init__(self, prot=None, id=None, name='', initializer=None, briefdescription=None,
detaileddescription=None, mixedclass_=None, content_=None):
supermod.enumvalueType.__init__(self, mixedclass_, content_)
self.initializer = None
def buildChildren(self, child_, nodeName_):
# Get text from <name> child and put it in self.name
if child_.nodeType == Node.ELEMENT_NODE and nodeName_ == 'name':
value_ = []
for text_ in child_.childNodes:
value_.append(text_.nodeValue)
valuestr_ = ''.join(value_)
self.name = valuestr_
elif child_.nodeType == Node.ELEMENT_NODE and nodeName_ == 'briefdescription':
obj_ = supermod.descriptionType.factory()
obj_.build(child_)
self.set_briefdescription(obj_)
elif child_.nodeType == Node.ELEMENT_NODE and nodeName_ == 'detaileddescription':
obj_ = supermod.descriptionType.factory()
obj_.build(child_)
self.set_detaileddescription(obj_)
elif child_.nodeType == Node.ELEMENT_NODE and nodeName_ == 'initializer':
childobj_ = supermod.linkedTextType.factory()
childobj_.build(child_)
obj_ = self.mixedclass_(MixedContainer.CategoryComplex, MixedContainer.TypeNone,
'initializer', childobj_)
self.set_initializer(obj_)
self.content_.append(obj_)
supermod.enumvalueType.subclass = enumvalueTypeSub
# end class enumvalueTypeSub
class templateparamlistTypeSub(supermod.templateparamlistType):
node_type = "templateparamlist"
def __init__(self, param=None):
supermod.templateparamlistType.__init__(self, param)
supermod.templateparamlistType.subclass = templateparamlistTypeSub
# end class templateparamlistTypeSub
class paramTypeSub(supermod.paramType):
node_type = "param"
def __init__(self, type_=None, declname='', defname='', array='', defval=None,
briefdescription=None):
supermod.paramType.__init__(self, type_, declname, defname, array, defval, briefdescription)
supermod.paramType.subclass = paramTypeSub
# end class paramTypeSub
class linkedTextTypeSub(supermod.linkedTextType):
node_type = "linkedtext"
def __init__(self, ref=None, mixedclass_=None, content_=None):
supermod.linkedTextType.__init__(self, mixedclass_, content_)
supermod.linkedTextType.subclass = linkedTextTypeSub
# end class linkedTextTypeSub
class graphTypeSub(supermod.graphType):
node_type = "graph"
def __init__(self, node=None):
supermod.graphType.__init__(self, node)
supermod.graphType.subclass = graphTypeSub
# end class graphTypeSub
class nodeTypeSub(supermod.nodeType):
node_type = "node"
def __init__(self, id=None, label='', link=None, childnode=None):
supermod.nodeType.__init__(self, id, label, link, childnode)
supermod.nodeType.subclass = nodeTypeSub
# end class nodeTypeSub
class childnodeTypeSub(supermod.childnodeType):
node_type = "childnode"
def __init__(self, relation=None, refid=None, edgelabel=None):
supermod.childnodeType.__init__(self, relation, refid, edgelabel)
supermod.childnodeType.subclass = childnodeTypeSub
# end class childnodeTypeSub
class linkTypeSub(supermod.linkType):
node_type = "link"
def __init__(self, refid=None, external=None, valueOf_=''):
supermod.linkType.__init__(self, refid, external)
supermod.linkType.subclass = linkTypeSub
# end class linkTypeSub
class listingTypeSub(supermod.listingType):
node_type = "listing"
def __init__(self, codeline=None):
supermod.listingType.__init__(self, codeline)
supermod.listingType.subclass = listingTypeSub
# end class listingTypeSub
class codelineTypeSub(supermod.codelineType):
node_type = "codeline"
def __init__(self, external=None, lineno=None, refkind=None, refid=None, highlight=None):
supermod.codelineType.__init__(self, external, lineno, refkind, refid, highlight)
supermod.codelineType.subclass = codelineTypeSub
# end class codelineTypeSub
class highlightTypeSub(supermod.highlightType):
node_type = "highlight"
def __init__(self, class_=None, sp=None, ref=None, mixedclass_=None, content_=None):
supermod.highlightType.__init__(self, mixedclass_, content_)
supermod.highlightType.subclass = highlightTypeSub
# end class highlightTypeSub
class referenceTypeSub(supermod.referenceType):
node_type = "reference"
def __init__(self, endline=None, startline=None, refid=None, compoundref=None, valueOf_='',
mixedclass_=None, content_=None):
supermod.referenceType.__init__(self, mixedclass_, content_)
supermod.referenceType.subclass = referenceTypeSub
# end class referenceTypeSub
class locationTypeSub(supermod.locationType):
node_type = "location"
def __init__(self, bodystart=None, line=None, bodyend=None, bodyfile=None, file=None,
valueOf_=''):
supermod.locationType.__init__(self, bodystart, line, bodyend, bodyfile, file)
supermod.locationType.subclass = locationTypeSub
# end class locationTypeSub
class docSect1TypeSub(supermod.docSect1Type):
node_type = "docsect1"
def __init__(self, id=None, title='', para=None, sect2=None, internal=None, mixedclass_=None,
content_=None):
supermod.docSect1Type.__init__(self, mixedclass_, content_)
supermod.docSect1Type.subclass = docSect1TypeSub
# end class docSect1TypeSub
class docSect2TypeSub(supermod.docSect2Type):
node_type = "docsect2"
def __init__(self, id=None, title='', para=None, sect3=None, internal=None, mixedclass_=None,
content_=None):
supermod.docSect2Type.__init__(self, mixedclass_, content_)
supermod.docSect2Type.subclass = docSect2TypeSub
# end class docSect2TypeSub
class docSect3TypeSub(supermod.docSect3Type):
node_type = "docsect3"
def __init__(self, id=None, title='', para=None, sect4=None, internal=None, mixedclass_=None,
content_=None):
supermod.docSect3Type.__init__(self, mixedclass_, content_)
supermod.docSect3Type.subclass = docSect3TypeSub
# end class docSect3TypeSub
class docSect4TypeSub(supermod.docSect4Type):
node_type = "docsect4"
def __init__(self, id=None, title='', para=None, internal=None, mixedclass_=None,
content_=None):
supermod.docSect4Type.__init__(self, mixedclass_, content_)
supermod.docSect4Type.subclass = docSect4TypeSub
# end class docSect4TypeSub
class docInternalTypeSub(supermod.docInternalType):
node_type = "docinternal"
def __init__(self, para=None, sect1=None, mixedclass_=None, content_=None):
supermod.docInternalType.__init__(self, mixedclass_, content_)
supermod.docInternalType.subclass = docInternalTypeSub
# end class docInternalTypeSub
class docInternalS1TypeSub(supermod.docInternalS1Type):
node_type = "docinternals1"
def __init__(self, para=None, sect2=None, mixedclass_=None, content_=None):
supermod.docInternalS1Type.__init__(self, mixedclass_, content_)
supermod.docInternalS1Type.subclass = docInternalS1TypeSub
# end class docInternalS1TypeSub
class docInternalS2TypeSub(supermod.docInternalS2Type):
node_type = "docinternals2"
def __init__(self, para=None, sect3=None, mixedclass_=None, content_=None):
supermod.docInternalS2Type.__init__(self, mixedclass_, content_)
supermod.docInternalS2Type.subclass = docInternalS2TypeSub
# end class docInternalS2TypeSub
class docInternalS3TypeSub(supermod.docInternalS3Type):
node_type = "docinternals3"
def __init__(self, para=None, sect3=None, mixedclass_=None, content_=None):
supermod.docInternalS3Type.__init__(self, mixedclass_, content_)
supermod.docInternalS3Type.subclass = docInternalS3TypeSub
# end class docInternalS3TypeSub
class docInternalS4TypeSub(supermod.docInternalS4Type):
node_type = "docinternals4"
def __init__(self, para=None, mixedclass_=None, content_=None):
supermod.docInternalS4Type.__init__(self, mixedclass_, content_)
supermod.docInternalS4Type.subclass = docInternalS4TypeSub
# end class docInternalS4TypeSub
class docURLLinkSub(supermod.docURLLink):
node_type = "docurllink"
def __init__(self, url=None, valueOf_='', mixedclass_=None, content_=None):
supermod.docURLLink.__init__(self, mixedclass_, content_)
supermod.docURLLink.subclass = docURLLinkSub
# end class docURLLinkSub
class docAnchorTypeSub(supermod.docAnchorType):
node_type = "docanchor"
def __init__(self, id=None, valueOf_='', mixedclass_=None, content_=None):
supermod.docAnchorType.__init__(self, mixedclass_, content_)
supermod.docAnchorType.subclass = docAnchorTypeSub
# end class docAnchorTypeSub
class docFormulaTypeSub(supermod.docFormulaType):
node_type = "docformula"
def __init__(self, id=None, valueOf_='', mixedclass_=None, content_=None):
supermod.docFormulaType.__init__(self, mixedclass_, content_)
supermod.docFormulaType.subclass = docFormulaTypeSub
# end class docFormulaTypeSub
class docIndexEntryTypeSub(supermod.docIndexEntryType):
node_type = "docindexentry"
def __init__(self, primaryie='', secondaryie=''):
supermod.docIndexEntryType.__init__(self, primaryie, secondaryie)
supermod.docIndexEntryType.subclass = docIndexEntryTypeSub
# end class docIndexEntryTypeSub
class docListTypeSub(supermod.docListType):
node_type = "doclist"
def __init__(self, listitem=None, subtype=""):
self.node_subtype = "itemized"
if subtype is not "":
self.node_subtype = subtype
supermod.docListType.__init__(self, listitem)
supermod.docListType.subclass = docListTypeSub
# end class docListTypeSub
class docListItemTypeSub(supermod.docListItemType):
node_type = "doclistitem"
def __init__(self, para=None):
supermod.docListItemType.__init__(self, para)
supermod.docListItemType.subclass = docListItemTypeSub
# end class docListItemTypeSub
class docSimpleSectTypeSub(supermod.docSimpleSectType):
node_type = "docsimplesect"
def __init__(self, kind=None, title=None, para=None):
supermod.docSimpleSectType.__init__(self, kind, title, para)
supermod.docSimpleSectType.subclass = docSimpleSectTypeSub
# end class docSimpleSectTypeSub
class docVarListEntryTypeSub(supermod.docVarListEntryType):
node_type = "docvarlistentry"
def __init__(self, term=None):
supermod.docVarListEntryType.__init__(self, term)
supermod.docVarListEntryType.subclass = docVarListEntryTypeSub
# end class docVarListEntryTypeSub
class docRefTextTypeSub(supermod.docRefTextType):
node_type = "docreftext"
def __init__(self, refid=None, kindref=None, external=None, valueOf_='', mixedclass_=None,
content_=None):
supermod.docRefTextType.__init__(self, mixedclass_, content_)
self.para = []
def buildChildren(self, child_, nodeName_):
supermod.docRefTextType.buildChildren(self, child_, nodeName_)
if child_.nodeType == Node.ELEMENT_NODE and nodeName_ == 'para':
obj_ = supermod.docParaType.factory()
obj_.build(child_)
self.para.append(obj_)
supermod.docRefTextType.subclass = docRefTextTypeSub
# end class docRefTextTypeSub
class docTableTypeSub(supermod.docTableType):
node_type = "doctable"
def __init__(self, rows=None, cols=None, row=None, caption=None):
supermod.docTableType.__init__(self, rows, cols, row, caption)
supermod.docTableType.subclass = docTableTypeSub
# end class docTableTypeSub
class docRowTypeSub(supermod.docRowType):
node_type = "docrow"
def __init__(self, entry=None):
supermod.docRowType.__init__(self, entry)
supermod.docRowType.subclass = docRowTypeSub
# end class docRowTypeSub
class docEntryTypeSub(supermod.docEntryType):
node_type = "docentry"
def __init__(self, thead=None, para=None):
supermod.docEntryType.__init__(self, thead, para)
supermod.docEntryType.subclass = docEntryTypeSub
# end class docEntryTypeSub
class docHeadingTypeSub(supermod.docHeadingType):
node_type = "docheading"
def __init__(self, level=None, valueOf_='', mixedclass_=None, content_=None):
supermod.docHeadingType.__init__(self, mixedclass_, content_)
def buildChildren(self, child_, nodeName_):
supermod.docHeadingType.buildChildren(self, child_, nodeName_)
# Account for styled content in the heading. This might need to be expanded to include other
# nodes as it seems from the xsd that headings can have a lot of different children but we
# really don't expect most of them to come up.
if child_.nodeType == Node.ELEMENT_NODE and (
nodeName_ == 'bold' or
nodeName_ == 'emphasis' or
nodeName_ == 'computeroutput' or
nodeName_ == 'subscript' or
nodeName_ == 'superscript' or
nodeName_ == 'center' or
nodeName_ == 'small'):
obj_ = supermod.docMarkupType.factory()
obj_.build(child_)
obj_.type_ = nodeName_
self.content_.append(obj_)
supermod.docHeadingType.subclass = docHeadingTypeSub
# end class docHeadingTypeSub
class docImageTypeSub(supermod.docImageType):
node_type = "docimage"
def __init__(self, width=None, type_=None, name=None, height=None, valueOf_='',
mixedclass_=None, content_=None):
supermod.docImageType.__init__(self, mixedclass_, content_)
supermod.docImageType.subclass = docImageTypeSub
# end class docImageTypeSub
class docDotFileTypeSub(supermod.docDotFileType):
node_type = "docdocfile"
def __init__(self, name=None, valueOf_='', mixedclass_=None, content_=None):
supermod.docDotFileType.__init__(self, mixedclass_, content_)
supermod.docDotFileType.subclass = docDotFileTypeSub
# end class docDotFileTypeSub
class docTocItemTypeSub(supermod.docTocItemType):
node_type = "doctocitem"
def __init__(self, id=None, valueOf_='', mixedclass_=None, content_=None):
supermod.docTocItemType.__init__(self, mixedclass_, content_)
supermod.docTocItemType.subclass = docTocItemTypeSub
# end class docTocItemTypeSub
class docTocListTypeSub(supermod.docTocListType):
node_type = "doctoclist"
def __init__(self, tocitem=None):
supermod.docTocListType.__init__(self, tocitem)
supermod.docTocListType.subclass = docTocListTypeSub
# end class docTocListTypeSub
class docLanguageTypeSub(supermod.docLanguageType):
node_type = "doclanguage"
def __init__(self, langid=None, para=None):
supermod.docLanguageType.__init__(self, langid, para)
supermod.docLanguageType.subclass = docLanguageTypeSub
# end class docLanguageTypeSub
class docParamListTypeSub(supermod.docParamListType):
node_type = "docparamlist"
def __init__(self, kind=None, parameteritem=None):
supermod.docParamListType.__init__(self, kind, parameteritem)
supermod.docParamListType.subclass = docParamListTypeSub
# end class docParamListTypeSub
class docParamListItemSub(supermod.docParamListItem):
node_type = "docparamlistitem"
def __init__(self, parameternamelist=None, parameterdescription=None):
supermod.docParamListItem.__init__(self, parameternamelist, parameterdescription)
supermod.docParamListItem.subclass = docParamListItemSub
# end class docParamListItemSub
class docParamNameListSub(supermod.docParamNameList):
node_type = "docparamnamelist"
def __init__(self, parametername=None):
supermod.docParamNameList.__init__(self, parametername)
supermod.docParamNameList.subclass = docParamNameListSub
# end class docParamNameListSub
class docParamNameSub(supermod.docParamName):
node_type = "docparamname"
def __init__(self, direction=None, ref=None, mixedclass_=None, content_=None):
supermod.docParamName.__init__(self, mixedclass_, content_)
supermod.docParamName.subclass = docParamNameSub
# end class docParamNameSub
class docXRefSectTypeSub(supermod.docXRefSectType):
node_type = "docxrefsect"
def __init__(self, id=None, xreftitle=None, xrefdescription=None):
supermod.docXRefSectType.__init__(self, id, xreftitle, xrefdescription)
supermod.docXRefSectType.subclass = docXRefSectTypeSub
# end class docXRefSectTypeSub
class docCopyTypeSub(supermod.docCopyType):
node_type = "doccopy"
def __init__(self, link=None, para=None, sect1=None, internal=None):
supermod.docCopyType.__init__(self, link, para, sect1, internal)
supermod.docCopyType.subclass = docCopyTypeSub
# end class docCopyTypeSub
class docCharTypeSub(supermod.docCharType):
node_type = "docchar"
def __init__(self, char=None, valueOf_=''):
supermod.docCharType.__init__(self, char)
supermod.docCharType.subclass = docCharTypeSub
# end class docCharTypeSub
class verbatimTypeSub(object):
"""
New node type. Structure is largely pillaged from other nodes in order to
match the set.
"""
node_type = "verbatim"
def __init__(self, valueOf_='', mixedclass_=None, content_=None):
if mixedclass_ is None:
self.mixedclass_ = MixedContainer
else:
self.mixedclass_ = mixedclass_
if content_ is None:
self.content_ = []
else:
self.content_ = content_
self.text = ""
def factory(*args, **kwargs):
return verbatimTypeSub(*args, **kwargs)
factory = staticmethod(factory)
def buildAttributes(self, attrs):
pass
def build(self, node_):
attrs = node_.attributes
self.buildAttributes(attrs)
self.valueOf_ = ''
for child_ in node_.childNodes:
nodeName_ = child_.nodeName.split(':')[-1]
self.buildChildren(child_, nodeName_)
def buildChildren(self, child_, nodeName_):
if child_.nodeType == Node.TEXT_NODE:
self.text += child_.nodeValue
class docParaTypeSub(supermod.docParaType):
node_type = "docpara"
def __init__(self, char=None, valueOf_=''):
supermod.docParaType.__init__(self, char)
self.parameterlist = []
self.simplesects = []
self.content = []
self.programlisting = []
self.images = []
def buildChildren(self, child_, nodeName_):
supermod.docParaType.buildChildren(self, child_, nodeName_)
if child_.nodeType == Node.TEXT_NODE:
obj_ = self.mixedclass_(MixedContainer.CategoryText,
MixedContainer.TypeNone, '', child_.nodeValue)
self.content.append(obj_)
elif child_.nodeType == Node.ELEMENT_NODE and nodeName_ == "ref":
obj_ = supermod.docRefTextType.factory()
obj_.build(child_)
self.content.append(obj_)
elif child_.nodeType == Node.ELEMENT_NODE and nodeName_ == 'parameterlist':
obj_ = supermod.docParamListType.factory()
obj_.build(child_)
self.parameterlist.append(obj_)
elif child_.nodeType == Node.ELEMENT_NODE and nodeName_ == 'simplesect':
obj_ = supermod.docSimpleSectType.factory()
obj_.build(child_)
self.simplesects.append(obj_)
elif child_.nodeType == Node.ELEMENT_NODE and nodeName_ == 'programlisting':
obj_ = supermod.listingType.factory()
obj_.build(child_)
self.programlisting.append(obj_)
elif child_.nodeType == Node.ELEMENT_NODE and nodeName_ == 'image':
obj_ = supermod.docImageType.factory()
obj_.build(child_)
self.images.append(obj_)
elif child_.nodeType == Node.ELEMENT_NODE and (
nodeName_ == 'bold' or
nodeName_ == 'emphasis' or
nodeName_ == 'computeroutput' or
nodeName_ == 'subscript' or
nodeName_ == 'superscript' or
nodeName_ == 'center' or
nodeName_ == 'small'):
obj_ = supermod.docMarkupType.factory()
obj_.build(child_)
obj_.type_ = nodeName_
self.content.append(obj_)
elif child_.nodeType == Node.ELEMENT_NODE and nodeName_ == 'verbatim':
childobj_ = verbatimTypeSub.factory()
childobj_.build(child_)
obj_ = self.mixedclass_(MixedContainer.CategoryComplex, MixedContainer.TypeNone,
'verbatim', childobj_)
self.content.append(obj_)
elif child_.nodeType == Node.ELEMENT_NODE and nodeName_ == 'formula':
childobj_ = docFormulaTypeSub.factory()
childobj_.build(child_)
obj_ = self.mixedclass_(MixedContainer.CategoryComplex, MixedContainer.TypeNone,
'formula', childobj_)
self.content.append(obj_)
elif child_.nodeType == Node.ELEMENT_NODE and nodeName_ == "itemizedlist":
obj_ = supermod.docListType.factory(subtype="itemized")
obj_.build(child_)
self.content.append(obj_)
elif child_.nodeType == Node.ELEMENT_NODE and nodeName_ == "orderedlist":
obj_ = supermod.docListType.factory(subtype="ordered")
obj_.build(child_)
self.content.append(obj_)
elif child_.nodeType == Node.ELEMENT_NODE and nodeName_ == 'heading':
obj_ = supermod.docHeadingType.factory()
obj_.build(child_)
self.content.append(obj_)
elif child_.nodeType == Node.ELEMENT_NODE and nodeName_ == 'ulink':
obj_ = supermod.docURLLink.factory()
obj_.build(child_)
self.content.append(obj_)
supermod.docParaType.subclass = docParaTypeSub
# end class docParaTypeSub
class docMarkupTypeSub(supermod.docMarkupType):
node_type = "docmarkup"
def __init__(self, valueOf_='', mixedclass_=None, content_=None):
supermod.docMarkupType.__init__(self, valueOf_, mixedclass_, content_)
self.type_ = None
def buildChildren(self, child_, nodeName_):
if child_.nodeType == Node.TEXT_NODE:
obj_ = self.mixedclass_(MixedContainer.CategoryText, MixedContainer.TypeNone, '',
child_.nodeValue)
self.content_.append(obj_)
elif child_.nodeType == Node.ELEMENT_NODE and nodeName_ == 'ref':
childobj_ = supermod.docRefTextType.factory()
childobj_.build(child_)
obj_ = self.mixedclass_(MixedContainer.CategoryComplex, MixedContainer.TypeNone, 'ref',
childobj_)
self.content_.append(obj_)
if child_.nodeType == Node.TEXT_NODE:
self.valueOf_ += child_.nodeValue
elif child_.nodeType == Node.CDATA_SECTION_NODE:
self.valueOf_ += '![CDATA[' + child_.nodeValue + ']]'
supermod.docMarkupType.subclass = docMarkupTypeSub
# end class docMarkupTypeSub
class docTitleTypeSub(supermod.docTitleType):
node_type = "doctitle"
def __init__(self, valueOf_='', mixedclass_=None, content_=None):
supermod.docTitleType.__init__(self, valueOf_, mixedclass_, content_)
self.type_ = None
supermod.docTitleType.subclass = docTitleTypeSub
# end class docTitleTypeSub
class ParseError(Exception):
pass
class FileIOError(Exception):
pass
def parse(inFilename):
try:
doc = minidom.parse(inFilename)
except IOError as e:
raise FileIOError(e)
except ExpatError as e:
raise ParseError(e)
rootNode = doc.documentElement
rootObj = supermod.DoxygenType.factory()
rootObj.build(rootNode)
return rootObj

File diff suppressed because it is too large Load Diff

View File

@ -1,63 +0,0 @@
#!/usr/bin/env python
"""
Generated Mon Feb 9 19:08:05 2009 by generateDS.py.
"""
from xml.dom import minidom
from xml.parsers.expat import ExpatError
from . import indexsuper as supermod
class DoxygenTypeSub(supermod.DoxygenType):
node_type = "doxygen"
def __init__(self, version=None, compound=None):
supermod.DoxygenType.__init__(self, version, compound)
supermod.DoxygenType.subclass = DoxygenTypeSub
# end class DoxygenTypeSub
class CompoundTypeSub(supermod.CompoundType):
node_type = "compound"
def __init__(self, kind=None, refid=None, name='', member=None):
supermod.CompoundType.__init__(self, kind, refid, name, member)
supermod.CompoundType.subclass = CompoundTypeSub
# end class CompoundTypeSub
class MemberTypeSub(supermod.MemberType):
node_type = "member"
def __init__(self, kind=None, refid=None, name=''):
supermod.MemberType.__init__(self, kind, refid, name)
supermod.MemberType.subclass = MemberTypeSub
# end class MemberTypeSub
class ParseError(Exception):
pass
class FileIOError(Exception):
pass
def parse(inFilename):
try:
doc = minidom.parse(inFilename)
except IOError as e:
raise FileIOError(e)
except ExpatError as e:
raise ParseError(e)
rootNode = doc.documentElement
rootObj = supermod.DoxygenType.factory()
rootObj.build(rootNode)
return rootObj

Binary file not shown.

View File

@ -1,362 +0,0 @@
#!/usr/bin/env python
#
# Generated Thu Jun 11 18:43:54 2009 by generateDS.py.
#
import sys
import getopt
from xml.dom import minidom
from xml.dom import Node
#
# User methods
#
# Calls to the methods in these classes are generated by generateDS.py.
# You can replace these methods by re-implementing the following class
# in a module named generatedssuper.py.
try:
from generatedssuper import GeneratedsSuper
except ImportError as exp:
class GeneratedsSuper:
def format_string(self, input_data, input_name=''):
return input_data
def format_integer(self, input_data, input_name=''):
return '%d' % input_data
def format_float(self, input_data, input_name=''):
return '%f' % input_data
def format_double(self, input_data, input_name=''):
return '%e' % input_data
def format_boolean(self, input_data, input_name=''):
return '%s' % input_data
#
# If you have installed IPython you can uncomment and use the following.
# IPython is available from http://ipython.scipy.org/.
#
## from IPython.Shell import IPShellEmbed
## args = ''
## ipshell = IPShellEmbed(args,
## banner = 'Dropping into IPython',
## exit_msg = 'Leaving Interpreter, back to program.')
# Then use the following line where and when you want to drop into the
# IPython shell:
# ipshell('<some message> -- Entering ipshell.\nHit Ctrl-D to exit')
#
# Globals
#
ExternalEncoding = 'ascii'
#
# Support/utility functions.
#
def showIndent(outfile, level):
for idx in range(level):
outfile.write(' ')
def quote_xml(inStr):
s1 = (isinstance(inStr, basestring) and inStr or
'%s' % inStr)
s1 = s1.replace('&', '&amp;')
s1 = s1.replace('<', '&lt;')
s1 = s1.replace('>', '&gt;')
return s1
def quote_attrib(inStr):
s1 = (isinstance(inStr, basestring) and inStr or
'%s' % inStr)
s1 = s1.replace('&', '&amp;')
s1 = s1.replace('<', '&lt;')
s1 = s1.replace('>', '&gt;')
if '"' in s1:
if "'" in s1:
s1 = '"%s"' % s1.replace('"', "&quot;")
else:
s1 = "'%s'" % s1
else:
s1 = '"%s"' % s1
return s1
def quote_python(inStr):
s1 = inStr
if s1.find("'") == -1:
if s1.find('\n') == -1:
return "'%s'" % s1
else:
return "'''%s'''" % s1
else:
if s1.find('"') != -1:
s1 = s1.replace('"', '\\"')
if s1.find('\n') == -1:
return '"%s"' % s1
else:
return '"""%s"""' % s1
class MixedContainer:
# Constants for category:
CategoryNone = 0
CategoryText = 1
CategorySimple = 2
CategoryComplex = 3
# Constants for content_type:
TypeNone = 0
TypeText = 1
TypeString = 2
TypeInteger = 3
TypeFloat = 4
TypeDecimal = 5
TypeDouble = 6
TypeBoolean = 7
def __init__(self, category, content_type, name, value):
self.category = category
self.content_type = content_type
self.name = name
self.value = value
def getCategory(self):
return self.category
def getContenttype(self, content_type):
return self.content_type
def getValue(self):
return self.value
def getName(self):
return self.name
class _MemberSpec(object):
def __init__(self, name='', data_type='', container=0):
self.name = name
self.data_type = data_type
self.container = container
def set_name(self, name): self.name = name
def get_name(self): return self.name
def set_data_type(self, data_type): self.data_type = data_type
def get_data_type(self): return self.data_type
def set_container(self, container): self.container = container
def get_container(self): return self.container
#
# Data representation classes.
#
class DoxygenType(GeneratedsSuper):
subclass = None
superclass = None
def __init__(self, version=None, compound=None):
self.version = version
if compound is None:
self.compound = []
else:
self.compound = compound
def factory(*args_, **kwargs_):
if DoxygenType.subclass:
return DoxygenType.subclass(*args_, **kwargs_)
else:
return DoxygenType(*args_, **kwargs_)
factory = staticmethod(factory)
def get_compound(self): return self.compound
def set_compound(self, compound): self.compound = compound
def add_compound(self, value): self.compound.append(value)
def insert_compound(self, index, value): self.compound[index] = value
def get_version(self): return self.version
def set_version(self, version): self.version = version
def hasContent_(self):
if (
self.compound is not None
):
return True
else:
return False
def build(self, node_):
attrs = node_.attributes
self.buildAttributes(attrs)
for child_ in node_.childNodes:
nodeName_ = child_.nodeName.split(':')[-1]
self.buildChildren(child_, nodeName_)
def buildAttributes(self, attrs):
if attrs.get('version'):
self.version = attrs.get('version').value
def buildChildren(self, child_, nodeName_):
if child_.nodeType == Node.ELEMENT_NODE and \
nodeName_ == 'compound':
obj_ = CompoundType.factory()
obj_.build(child_)
self.compound.append(obj_)
# end class DoxygenType
class CompoundType(GeneratedsSuper):
subclass = None
superclass = None
def __init__(self, kind=None, refid=None, name=None, member=None):
self.kind = kind
self.refid = refid
self.name = name
if member is None:
self.member = []
else:
self.member = member
def factory(*args_, **kwargs_):
if CompoundType.subclass:
return CompoundType.subclass(*args_, **kwargs_)
else:
return CompoundType(*args_, **kwargs_)
factory = staticmethod(factory)
def get_name(self): return self.name
def set_name(self, name): self.name = name
def get_member(self): return self.member
def set_member(self, member): self.member = member
def add_member(self, value): self.member.append(value)
def insert_member(self, index, value): self.member[index] = value
def get_kind(self): return self.kind
def set_kind(self, kind): self.kind = kind
def get_refid(self): return self.refid
def set_refid(self, refid): self.refid = refid
def build(self, node_):
attrs = node_.attributes
self.buildAttributes(attrs)
for child_ in node_.childNodes:
nodeName_ = child_.nodeName.split(':')[-1]
self.buildChildren(child_, nodeName_)
def buildAttributes(self, attrs):
if attrs.get('kind'):
self.kind = attrs.get('kind').value
if attrs.get('refid'):
self.refid = attrs.get('refid').value
def buildChildren(self, child_, nodeName_):
if child_.nodeType == Node.ELEMENT_NODE and \
nodeName_ == 'name':
name_ = ''
for text__content_ in child_.childNodes:
name_ += text__content_.nodeValue
self.name = name_
elif child_.nodeType == Node.ELEMENT_NODE and \
nodeName_ == 'member':
obj_ = MemberType.factory()
obj_.build(child_)
self.member.append(obj_)
# end class CompoundType
class MemberType(GeneratedsSuper):
subclass = None
superclass = None
def __init__(self, kind=None, refid=None, name=None):
self.kind = kind
self.refid = refid
self.name = name
def factory(*args_, **kwargs_):
if MemberType.subclass:
return MemberType.subclass(*args_, **kwargs_)
else:
return MemberType(*args_, **kwargs_)
factory = staticmethod(factory)
def get_name(self): return self.name
def set_name(self, name): self.name = name
def get_kind(self): return self.kind
def set_kind(self, kind): self.kind = kind
def get_refid(self): return self.refid
def set_refid(self, refid): self.refid = refid
def hasContent_(self):
if (
self.name is not None
):
return True
else:
return False
def build(self, node_):
attrs = node_.attributes
self.buildAttributes(attrs)
for child_ in node_.childNodes:
nodeName_ = child_.nodeName.split(':')[-1]
self.buildChildren(child_, nodeName_)
def buildAttributes(self, attrs):
if attrs.get('kind'):
self.kind = attrs.get('kind').value
if attrs.get('refid'):
self.refid = attrs.get('refid').value
def buildChildren(self, child_, nodeName_):
if child_.nodeType == Node.ELEMENT_NODE and \
nodeName_ == 'name':
name_ = ''
for text__content_ in child_.childNodes:
name_ += text__content_.nodeValue
self.name = name_
# end class MemberType
USAGE_TEXT = """
Usage: python <Parser>.py [ -s ] <in_xml_file>
Options:
-s Use the SAX parser, not the minidom parser.
"""
def usage():
print(USAGE_TEXT)
sys.exit(1)
def parse(inFileName):
doc = minidom.parse(inFileName)
rootNode = doc.documentElement
rootObj = DoxygenType.factory()
rootObj.build(rootNode)
# Enable Python to collect the space used by the DOM.
doc = None
sys.stdout.write('<?xml version="1.0" ?>\n')
rootObj.export(sys.stdout, 0, name_="doxygenindex",
namespacedef_='')
return rootObj
def parseString(inString):
doc = minidom.parseString(inString)
rootNode = doc.documentElement
rootObj = DoxygenType.factory()
rootObj.build(rootNode)
# Enable Python to collect the space used by the DOM.
doc = None
sys.stdout.write('<?xml version="1.0" ?>\n')
rootObj.export(sys.stdout, 0, name_="doxygenindex",
namespacedef_='')
return rootObj
def parseLiteral(inFileName):
doc = minidom.parse(inFileName)
rootNode = doc.documentElement
rootObj = DoxygenType.factory()
rootObj.build(rootNode)
# Enable Python to collect the space used by the DOM.
doc = None
sys.stdout.write('from index import *\n\n')
sys.stdout.write('rootObj = doxygenindex(\n')
rootObj.exportLiteral(sys.stdout, 0, name_="doxygenindex")
sys.stdout.write(')\n')
return rootObj
def main():
args = sys.argv[1:]
if len(args) == 1:
parse(args[0])
else:
usage()
if __name__ == '__main__':
main()
#import pdb
#pdb.run('main()')

View File

@ -1,87 +0,0 @@
AUTOCFG_TEMPLATE = r"""
PROJECT_NAME = "{project_name}"
OUTPUT_DIRECTORY = {output_dir}
GENERATE_LATEX = NO
GENERATE_MAN = NO
GENERATE_RTF = NO
CASE_SENSE_NAMES = NO
INPUT = {input}
ENABLE_PREPROCESSING = YES
QUIET = YES
JAVADOC_AUTOBRIEF = YES
JAVADOC_AUTOBRIEF = NO
GENERATE_HTML = NO
GENERATE_XML = YES
ALIASES = "rst=\verbatim embed:rst"
ALIASES += "endrst=\endverbatim"
""".strip()
class ProjectData(object):
"Simple handler for the files and project_info for each project"
def __init__(self, auto_project_info, files):
self.auto_project_info = auto_project_info
self.files = files
class AutoDoxygenProcessHandle(object):
def __init__(self, path_handler, run_process, write_file, project_info_factory):
self.path_handler = path_handler
self.run_process = run_process
self.write_file = write_file
self.project_info_factory = project_info_factory
def generate_xml(self, app):
project_files = {}
# First collect together all the files which need to be doxygen processed for each project
for project_name, file_structure in app.config.breathe_projects_source.items():
folder = file_structure[0]
contents = file_structure[1]
auto_project_info = self.project_info_factory.create_auto_project_info(
project_name, folder)
project_files[project_name] = ProjectData(auto_project_info, contents)
# Iterate over the projects and generate doxygen xml output for the files for each one into
# a directory in the Sphinx build area
for project_name, data in project_files.items():
project_path = self.process(data.auto_project_info, data.files)
project_info = data.auto_project_info.create_project_info(project_path)
self.project_info_factory.store_project_info_for_auto(project_name, project_info)
def process(self, auto_project_info, files):
name = auto_project_info.name()
cfgfile = "%s.cfg" % name
full_paths = map(lambda x: auto_project_info.abs_path_to_source_file(x), files)
cfg = AUTOCFG_TEMPLATE.format(
project_name=name,
output_dir=name,
input=" ".join(full_paths)
)
build_dir = self.path_handler.join(
auto_project_info.build_dir(),
"breathe",
"doxygen"
)
self.write_file(build_dir, cfgfile, cfg)
self.run_process(['doxygen', cfgfile], cwd=build_dir)
return self.path_handler.join(build_dir, name, "xml")

Binary file not shown.

View File

@ -1,306 +0,0 @@
from .exception import BreatheError
import os
class ProjectError(BreatheError):
pass
class NoDefaultProjectError(ProjectError):
pass
class AutoProjectInfo(object):
"""Created as a temporary step in the automatic xml generation process"""
def __init__(
self,
name,
source_path,
build_dir,
reference,
source_dir,
config_dir,
domain_by_extension,
domain_by_file_pattern,
match
):
self._name = name
self._source_path = source_path
self._build_dir = build_dir
self._reference = reference
self._source_dir = source_dir
self._config_dir = config_dir
self._domain_by_extension = domain_by_extension
self._domain_by_file_pattern = domain_by_file_pattern
self._match = match
def name(self):
return self._name
def build_dir(self):
return self._build_dir
def abs_path_to_source_file(self, file_):
"""
Returns full path to the provide file assuming that the provided path is relative to the
projects conf.py directory as specified in the breathe_projects_source config variable.
"""
# os.path.join does the appropriate handling if _source_path is an absolute path
return os.path.join(self._config_dir, self._source_path, file_)
def create_project_info(self, project_path):
"""Creates a proper ProjectInfo object based on the information in this AutoProjectInfo"""
return ProjectInfo(
self._name,
project_path,
self._source_path,
self._reference,
self._source_dir,
self._config_dir,
self._domain_by_extension,
self._domain_by_file_pattern,
self._match
)
class ProjectInfo(object):
def __init__(
self,
name,
path,
source_path,
reference,
source_dir,
config_dir,
domain_by_extension,
domain_by_file_pattern,
match
):
self._name = name
self._project_path = path
self._source_path = source_path
self._reference = reference
self._source_dir = source_dir
self._config_dir = config_dir
self._domain_by_extension = domain_by_extension
self._domain_by_file_pattern = domain_by_file_pattern
self._match = match
def name(self):
return self._name
def project_path(self):
return self._project_path
def source_path(self):
return self._source_path
def relative_path_to_xml_file(self, file_):
"""
Returns relative path from Sphinx documentation top-level source directory to the specified
file assuming that the specified file is a path relative to the doxygen xml output
directory.
"""
# os.path.join does the appropriate handling if _project_path is an absolute path
full_xml_project_path = os.path.join(self._config_dir, self._project_path, file_)
return os.path.relpath(
full_xml_project_path,
self._source_dir
)
def sphinx_abs_path_to_file(self, file_):
"""
Prepends os.path.sep to the value returned by relative_path_to_file.
This is to match Sphinx's concept of an absolute path which starts from the top-level source
directory of the project.
"""
return os.path.sep + self.relative_path_to_xml_file(file_)
def reference(self):
return self._reference
def domain_for_file(self, file_):
domain = ""
extension = file_.split(".")[-1]
try:
domain = self._domain_by_extension[extension]
except KeyError:
pass
for pattern, pattern_domain in self._domain_by_file_pattern.items():
if self._match(file_, pattern):
domain = pattern_domain
return domain
class ProjectInfoFactory(object):
def __init__(self, source_dir, build_dir, config_dir, match):
self.source_dir = source_dir
self.build_dir = build_dir
self.config_dir = config_dir
self.match = match
self.projects = {}
self.default_project = None
self.domain_by_extension = {}
self.domain_by_file_pattern = {}
self.project_count = 0
self.project_info_store = {}
self.project_info_for_auto_store = {}
self.auto_project_info_store = {}
def update(
self,
projects,
default_project,
domain_by_extension,
domain_by_file_pattern,
projects_source,
build_dir
):
self.projects = projects
self.default_project = default_project
self.domain_by_extension = domain_by_extension
self.domain_by_file_pattern = domain_by_file_pattern
self.projects_source = projects_source
# If the breathe config values has a non-empty value for build_dir then use that otherwise
# stick with the default
if build_dir:
self.build_dir = build_dir
def default_path(self):
if not self.default_project:
raise NoDefaultProjectError(
"No breathe_default_project config setting to fall back on "
"for directive with no 'project' or 'path' specified."
)
try:
return self.projects[self.default_project]
except KeyError:
raise ProjectError(
("breathe_default_project value '%s' does not seem to be a valid key for the "
"breathe_projects dictionary") % self.default_project
)
def create_project_info(self, options):
name = ""
if "project" in options:
try:
path = self.projects[options["project"]]
name = options["project"]
except KeyError:
raise ProjectError("Unable to find project '%s' in breathe_projects dictionary"
% options["project"])
elif "path" in options:
path = options["path"]
else:
path = self.default_path()
try:
return self.project_info_store[path]
except KeyError:
reference = name
if not name:
name = "project%s" % self.project_count
reference = path
self.project_count += 1
project_info = ProjectInfo(
name,
path,
"NoSourcePath",
reference,
self.source_dir,
self.config_dir,
self.domain_by_extension,
self.domain_by_file_pattern,
self.match
)
self.project_info_store[path] = project_info
return project_info
def store_project_info_for_auto(self, name, project_info):
"""Stores the project info by name for later extraction by the auto directives.
Stored separately to the non-auto project info objects as they should never overlap.
"""
self.project_info_for_auto_store[name] = project_info
def retrieve_project_info_for_auto(self, options):
"""Retrieves the project info by name for later extraction by the auto directives.
Looks for the 'project' entry in the options dictionary. This is a less than ideal API but
it is designed to match the use of 'create_project_info' above for which it makes much more
sense.
"""
name = options.get('project', self.default_project)
if name is None:
raise NoDefaultProjectError(
"No breathe_default_project config setting to fall back on "
"for directive with no 'project' or 'path' specified."
)
return self.project_info_for_auto_store[name]
def create_auto_project_info(self, name, source_path):
key = source_path
try:
return self.auto_project_info_store[key]
except KeyError:
reference = name
if not name:
name = "project%s" % self.project_count
reference = source_path
self.project_count += 1
auto_project_info = AutoProjectInfo(
name,
source_path,
self.build_dir,
reference,
self.source_dir,
self.config_dir,
self.domain_by_extension,
self.domain_by_file_pattern,
self.match
)
self.auto_project_info_store[key] = auto_project_info
return auto_project_info

Binary file not shown.

Binary file not shown.

View File

@ -1,2 +0,0 @@

View File

@ -1,383 +0,0 @@
from .base import Renderer, RenderContext
from . import index as indexrenderer
from . import compound as compoundrenderer
from docutils import nodes
import textwrap
class RstContentCreator(object):
def __init__(self, list_type, dedent):
self.list_type = list_type
self.dedent = dedent
def __call__(self, text):
# Remove the first line which is "embed:rst[:leading-asterisk]"
text = "\n".join(text.split(u"\n")[1:])
# Remove starting whitespace
text = self.dedent(text)
# Inspired by autodoc.py in Sphinx
result = self.list_type()
for line in text.split("\n"):
result.append(line, "<breathe>")
return result
class UnicodeRenderer(Renderer):
def render(self):
# Skip any nodes that are pure whitespace
# Probably need a better way to do this as currently we're only doing
# it skip whitespace between higher-level nodes, but this will also
# skip any pure whitespace entries in actual content nodes
#
# We counter that second issue slightly by allowing through single white spaces
#
if self.data_object.strip():
return [self.node_factory.Text(self.data_object)]
elif self.data_object == unicode(" "):
return [self.node_factory.Text(self.data_object)]
else:
return []
class NullRenderer(Renderer):
def __init__(self):
pass
def render(self):
return []
class DoxygenToRstRendererFactory(object):
def __init__(
self,
node_type,
renderers,
renderer_factory_creator,
node_factory,
project_info,
state,
document,
rst_content_creator,
filter_,
target_handler,
domain_directive_factory
):
self.node_type = node_type
self.node_factory = node_factory
self.project_info = project_info
self.renderers = renderers
self.renderer_factory_creator = renderer_factory_creator
self.state = state
self.document = document
self.rst_content_creator = rst_content_creator
self.filter_ = filter_
self.target_handler = target_handler
self.domain_directive_factory = domain_directive_factory
def create_renderer(
self,
context
):
parent_data_object = context.node_stack[1]
data_object = context.node_stack[0]
if not self.filter_.allow(context.node_stack):
return NullRenderer()
child_renderer_factory = self.renderer_factory_creator.create_child_factory(
self.project_info,
data_object,
self
)
try:
node_type = data_object.node_type
except AttributeError as e:
# Horrible hack to silence errors on filtering unicode objects
# until we fix the parsing
if type(data_object) == unicode:
node_type = "unicode"
else:
raise e
Renderer = self.renderers[node_type]
common_args = [
self.project_info,
context,
child_renderer_factory,
self.node_factory,
self.state,
self.document,
self.target_handler,
self.domain_directive_factory
]
if node_type == "docmarkup":
creator = self.node_factory.inline
if data_object.type_ == "emphasis":
creator = self.node_factory.emphasis
elif data_object.type_ == "computeroutput":
creator = self.node_factory.literal
elif data_object.type_ == "bold":
creator = self.node_factory.strong
elif data_object.type_ == "superscript":
creator = self.node_factory.superscript
elif data_object.type_ == "subscript":
creator = self.node_factory.subscript
elif data_object.type_ == "center":
print("Warning: does not currently handle 'center' text display")
elif data_object.type_ == "small":
print("Warning: does not currently handle 'small' text display")
return Renderer(
creator,
*common_args
)
if node_type == "verbatim":
return Renderer(
self.rst_content_creator,
*common_args
)
if node_type == "compound":
kind = data_object.kind
if kind in ["file", "dir", "page", "example", "group"]:
return Renderer(indexrenderer.FileRenderer, *common_args)
class_ = indexrenderer.CompoundTypeSubRenderer
# For compound node types Renderer is CreateCompoundTypeSubRenderer
# as defined below. This could be cleaner
return Renderer(
class_,
*common_args
)
if node_type == "memberdef":
if data_object.kind in ("function", "slot") or (data_object.kind == 'friend' and data_object.argsstring):
Renderer = compoundrenderer.FuncMemberDefTypeSubRenderer
elif data_object.kind == "enum":
Renderer = compoundrenderer.EnumMemberDefTypeSubRenderer
elif data_object.kind == "typedef":
Renderer = compoundrenderer.TypedefMemberDefTypeSubRenderer
elif data_object.kind == "variable":
Renderer = compoundrenderer.VariableMemberDefTypeSubRenderer
elif data_object.kind == "define":
Renderer = compoundrenderer.DefineMemberDefTypeSubRenderer
if node_type == "param":
return Renderer(
parent_data_object.node_type != "templateparamlist",
*common_args
)
if node_type == "docsimplesect":
if data_object.kind == "par":
Renderer = compoundrenderer.ParDocSimpleSectTypeSubRenderer
return Renderer(
*common_args
)
class CreateCompoundTypeSubRenderer(object):
def __init__(self, parser_factory):
self.parser_factory = parser_factory
def __call__(self, class_, project_info, *args):
compound_parser = self.parser_factory.create_compound_parser(project_info)
return class_(compound_parser, project_info, *args)
class CreateRefTypeSubRenderer(object):
def __init__(self, parser_factory):
self.parser_factory = parser_factory
def __call__(self, project_info, *args):
compound_parser = self.parser_factory.create_compound_parser(project_info)
return compoundrenderer.RefTypeSubRenderer(compound_parser, project_info, *args)
class DoxygenToRstRendererFactoryCreator(object):
def __init__(
self,
node_factory,
parser_factory,
domain_directive_factory,
rst_content_creator,
project_info
):
self.node_factory = node_factory
self.parser_factory = parser_factory
self.domain_directive_factory = domain_directive_factory
self.rst_content_creator = rst_content_creator
self.project_info = project_info
def create_factory(self, node_stack, state, document, filter_, target_handler):
data_object = node_stack[0]
renderers = {
"doxygen" : indexrenderer.DoxygenTypeSubRenderer,
"compound" : CreateCompoundTypeSubRenderer(self.parser_factory),
"doxygendef" : compoundrenderer.DoxygenTypeSubRenderer,
"compounddef" : compoundrenderer.CompoundDefTypeSubRenderer,
"sectiondef" : compoundrenderer.SectionDefTypeSubRenderer,
"memberdef" : compoundrenderer.MemberDefTypeSubRenderer,
"enumvalue" : compoundrenderer.EnumvalueTypeSubRenderer,
"linkedtext" : compoundrenderer.LinkedTextTypeSubRenderer,
"description" : compoundrenderer.DescriptionTypeSubRenderer,
"param" : compoundrenderer.ParamTypeSubRenderer,
"docreftext" : compoundrenderer.DocRefTextTypeSubRenderer,
"docheading" : compoundrenderer.DocHeadingTypeSubRenderer,
"docpara" : compoundrenderer.DocParaTypeSubRenderer,
"docmarkup" : compoundrenderer.DocMarkupTypeSubRenderer,
"docparamlist" : compoundrenderer.DocParamListTypeSubRenderer,
"docparamlistitem" : compoundrenderer.DocParamListItemSubRenderer,
"docparamnamelist" : compoundrenderer.DocParamNameListSubRenderer,
"docparamname" : compoundrenderer.DocParamNameSubRenderer,
"docsect1" : compoundrenderer.DocSect1TypeSubRenderer,
"docsimplesect" : compoundrenderer.DocSimpleSectTypeSubRenderer,
"doctitle" : compoundrenderer.DocTitleTypeSubRenderer,
"docformula" : compoundrenderer.DocForumlaTypeSubRenderer,
"docimage" : compoundrenderer.DocImageTypeSubRenderer,
"docurllink" : compoundrenderer.DocURLLinkSubRenderer,
"listing" : compoundrenderer.ListingTypeSubRenderer,
"codeline" : compoundrenderer.CodeLineTypeSubRenderer,
"highlight" : compoundrenderer.HighlightTypeSubRenderer,
"templateparamlist" : compoundrenderer.TemplateParamListRenderer,
"inc" : compoundrenderer.IncTypeSubRenderer,
"ref" : CreateRefTypeSubRenderer(self.parser_factory),
"verbatim" : compoundrenderer.VerbatimTypeSubRenderer,
"mixedcontainer" : compoundrenderer.MixedContainerRenderer,
"unicode" : UnicodeRenderer,
"doclist": compoundrenderer.DocListTypeSubRenderer,
"doclistitem": compoundrenderer.DocListItemTypeSubRenderer,
}
try:
node_type = data_object.node_type
except AttributeError as e:
# Horrible hack to silence errors on filtering unicode objects
# until we fix the parsing
if type(data_object) == unicode:
node_type = "unicode"
else:
raise e
return DoxygenToRstRendererFactory(
"root",
renderers,
self,
self.node_factory,
self.project_info,
state,
document,
self.rst_content_creator,
filter_,
target_handler,
self.domain_directive_factory
)
def create_child_factory( self, project_info, data_object, parent_renderer_factory ):
try:
node_type = data_object.node_type
except AttributeError as e:
# Horrible hack to silence errors on filtering unicode objects
# until we fix the parsing
if type(data_object) == unicode:
node_type = "unicode"
else:
raise e
return DoxygenToRstRendererFactory(
node_type,
parent_renderer_factory.renderers,
self,
self.node_factory,
parent_renderer_factory.project_info,
parent_renderer_factory.state,
parent_renderer_factory.document,
self.rst_content_creator,
parent_renderer_factory.filter_,
parent_renderer_factory.target_handler,
parent_renderer_factory.domain_directive_factory
)
# FactoryFactoryFactory. Ridiculous but necessary.
class DoxygenToRstRendererFactoryCreatorConstructor(object):
def __init__(
self,
node_factory,
parser_factory,
domain_directive_factory,
rst_content_creator
):
self.node_factory = node_factory
self.parser_factory = parser_factory
self.domain_directive_factory = domain_directive_factory
self.rst_content_creator = rst_content_creator
def create_factory_creator(self, project_info, document, options, target_handler):
return DoxygenToRstRendererFactoryCreator(
self.node_factory,
self.parser_factory,
self.domain_directive_factory,
self.rst_content_creator,
project_info,
)
def format_parser_error(name, error, filename, state, lineno, do_unicode_warning):
warning = '%s: Unable to parse xml file "%s". ' % (name, filename)
explanation = 'Reported error: %s. ' % error
unicode_explanation_text = ""
unicode_explanation = []
if do_unicode_warning:
unicode_explanation_text = textwrap.dedent("""
Parsing errors are often due to unicode errors associated with the encoding of the original
source files. Doxygen propagates invalid characters from the input source files to the
output xml.""").strip().replace("\n", " ")
unicode_explanation = [nodes.paragraph("", "", nodes.Text(unicode_explanation_text))]
return [nodes.warning("",
nodes.paragraph("", "", nodes.Text(warning)),
nodes.paragraph("", "", nodes.Text(explanation)),
*unicode_explanation
),
state.document.reporter.warning(warning + explanation + unicode_explanation_text, line=lineno)
]

View File

@ -1,127 +0,0 @@
class Renderer(object):
def __init__(self,
project_info,
context,
renderer_factory,
node_factory,
state,
document,
target_handler,
domain_directive_factory,
):
self.project_info = project_info
self.context = context
self.data_object = context.node_stack[0]
self.renderer_factory = renderer_factory
self.node_factory = node_factory
self.state = state
self.document = document
self.target_handler = target_handler
self.domain_directive_factory = domain_directive_factory
if self.context.domain == '':
self.context.domain = self.get_domain()
def get_domain(self):
"""Returns the domain for the current node."""
def get_filename(node):
"""Returns the name of a file where the declaration represented by node is located."""
try:
return node.location.file
except AttributeError:
return None
node_stack = self.context.node_stack
node = node_stack[0]
# An enumvalue node doesn't have location, so use its parent node for detecting the domain instead.
if type(node) == unicode or node.node_type == "enumvalue":
node = node_stack[1]
filename = get_filename(node)
if not filename and node.node_type == "compound":
file_data = self.compound_parser.parse(node.refid)
filename = get_filename(file_data.compounddef)
return self.project_info.domain_for_file(filename) if filename else ''
def get_fully_qualified_name(self):
names = []
node_stack = self.context.node_stack
node = node_stack[0]
if node.node_type == 'enumvalue':
names.append(node.name)
# Skip the name of the containing enum because it is not a part of the fully qualified name.
node_stack = node_stack[2:]
# If the node is a namespace, use its name because namespaces are skipped in the main loop.
if node.node_type == 'compound' and node.kind == 'namespace':
names.append(node.name)
for node in node_stack:
if node.node_type == 'ref' and len(names) == 0:
return node.valueOf_
if (node.node_type == 'compound' and node.kind not in ['file', 'namespace']) or \
node.node_type == 'memberdef':
# We skip the 'file' entries because the file name doesn't form part of the
# qualified name for the identifier. We skip the 'namespace' entries because if we
# find an object through the namespace 'compound' entry in the index.xml then we'll
# also have the 'compounddef' entry in our node stack and we'll get it from that. We
# need the 'compounddef' entry because if we find the object through the 'file'
# entry in the index.xml file then we need to get the namespace name from somewhere
names.insert(0, node.name)
if (node.node_type == 'compounddef' and node.kind == 'namespace'):
# Nested namespaces include their parent namespace(s) in compoundname. ie,
# compoundname is 'foo::bar' instead of just 'bar' for namespace 'bar' nested in
# namespace 'foo'. We need full compoundname because node_stack doesn't necessarily
# include parent namespaces and we stop here in case it does.
names.insert(0, node.compoundname)
break
return '::'.join(names)
def create_template_node(self, decl):
"""Creates a node for the ``template <...>`` part of the declaration."""
if not decl.templateparamlist:
return None
context = self.context.create_child_context(decl.templateparamlist)
renderer = self.renderer_factory.create_renderer(context)
nodes = [self.node_factory.Text("template <")]
nodes.extend(renderer.render())
nodes.append(self.node_factory.Text(">"))
signode = self.node_factory.desc_signature()
signode.extend(nodes)
return signode
def run_domain_directive(self, kind, names):
domain_directive = self.renderer_factory.domain_directive_factory.create(
self.context.domain, [kind, names] + self.context.directive_args[2:])
# Translate Breathe's no-link option into the standard noindex option.
if 'no-link' in self.context.directive_args[2]:
domain_directive.options['noindex'] = True
nodes = domain_directive.run()
# Filter out outer class names if we are rendering a member as a part of a class content.
signode = nodes[1].children[0]
if len(names) > 0 and self.context.child:
signode.children = [n for n in signode.children if not n.tagname == 'desc_addname']
return nodes
class RenderContext(object):
def __init__(self, node_stack, mask_factory, directive_args, domain='', child=False):
self.node_stack = node_stack
self.mask_factory = mask_factory
self.directive_args = directive_args
self.domain = domain
self.child = child
def create_child_context(self, data_object):
node_stack = self.node_stack[:]
node_stack.insert(0, self.mask_factory.mask(data_object))
return RenderContext(node_stack, self.mask_factory, self.directive_args, self.domain, True)

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@ -1,115 +0,0 @@
from .base import Renderer
class DoxygenTypeSubRenderer(Renderer):
def render(self):
nodelist = []
# Process all the compound children
for compound in self.data_object.get_compound():
context = self.context.create_child_context(compound)
compound_renderer = self.renderer_factory.create_renderer(context)
nodelist.extend(compound_renderer.render())
return nodelist
class CompoundRenderer(Renderer):
"""Base class for CompoundTypeSubRenderer and RefTypeSubRenderer."""
def __init__(self, compound_parser, render_empty_node, *args):
self.compound_parser = compound_parser
self.render_empty_node = render_empty_node
Renderer.__init__(self, *args)
def create_doxygen_target(self):
"""Can be overridden to create a target node which uses the doxygen refid information
which can be used for creating links between internal doxygen elements.
The default implementation should suffice most of the time.
"""
refid = "%s%s" % (self.project_info.name(), self.data_object.refid)
return self.target_handler.create_target(refid)
def render_signature(self, file_data, doxygen_target):
# Defer to domains specific directive.
name, kind = self.get_node_info(file_data)
self.context.directive_args[1] = [self.get_fully_qualified_name()]
nodes = self.run_domain_directive(kind, self.context.directive_args[1])
node = nodes[1]
signode, contentnode = node.children
# The cpp domain in Sphinx doesn't support structs at the moment, so change the text from "class "
# to the correct kind which can be "class " or "struct ".
signode[0] = self.node_factory.desc_annotation(kind + ' ', kind + ' ')
# Check if there is template information and format it as desired
template_signode = self.create_template_node(file_data.compounddef)
if template_signode:
node.insert(0, template_signode)
node.children[0].insert(0, doxygen_target)
return nodes, contentnode
def render(self):
# Read in the corresponding xml file and process
file_data = self.compound_parser.parse(self.data_object.refid)
parent_context = self.context.create_child_context(file_data)
data_renderer = self.renderer_factory.create_renderer(parent_context)
rendered_data = data_renderer.render()
if not rendered_data and not self.render_empty_node:
return []
file_data = parent_context.node_stack[0]
new_context = parent_context.create_child_context(file_data.compounddef)
nodes, contentnode = self.render_signature(file_data, self.create_doxygen_target())
if file_data.compounddef.includes:
for include in file_data.compounddef.includes:
context = new_context.create_child_context(include)
renderer = self.renderer_factory.create_renderer(context)
contentnode.extend(renderer.render())
contentnode.extend(rendered_data)
return nodes
class CompoundTypeSubRenderer(CompoundRenderer):
def __init__(self, compound_parser, *args):
CompoundRenderer.__init__(self, compound_parser, True, *args)
def get_node_info(self, file_data):
return self.data_object.name, self.data_object.kind
class FileRenderer(CompoundTypeSubRenderer):
def render_signature(self, file_data, doxygen_target):
# Build targets for linking
targets = []
targets.extend(doxygen_target)
title_signode = self.node_factory.desc_signature()
title_signode.extend(targets)
# Set up the title
name, kind = self.get_node_info(file_data)
title_signode.append(self.node_factory.emphasis(text=kind))
title_signode.append(self.node_factory.Text(" "))
title_signode.append(self.node_factory.desc_name(text=name))
contentnode = self.node_factory.desc_content()
node = self.node_factory.desc()
node.document = self.state.document
node['objtype'] = kind
node.append(title_signode)
node.append(contentnode)
return [node], contentnode

View File

@ -1,62 +0,0 @@
"""
Masks
=====
Masks are related to filters. Filters can block the processing of particular parts of the xml
hierarchy but they can only work on node level. If the part of the xml hierarchy that you want to
filter out is read in as an instance of one of the classes in parser/doxygen/*.py then you can use
the filters. However, if you want to filter out an attribute from one of the nodes (and some of the
xml child nodes are read in as attributes on their parents) then you can't use a filter.
We introduce the Mask's to fulfil this need. The masks are designed to be applied to a particular
node type and to limit the access to particular attributes on the node. For example, then
NoParameterNamesMask wraps a node a returns all its standard attributes but returns None for the
'declname' and 'defname' attributes.
Currently the Mask functionality is only used for the text signature rendering for doing function
matching.
"""
class NoParameterNamesMask(object):
def __init__(self, data_object):
self.data_object = data_object
def __getattr__(self, attr):
if attr in ['declname', 'defname', 'defval']:
return None
return getattr(self.data_object, attr)
class MaskFactory(object):
def __init__(self, lookup):
self.lookup = lookup
def mask(self, data_object):
try:
node_type = data_object.node_type
except AttributeError as e:
# Horrible hack to silence errors on filtering unicode objects
# until we fix the parsing
if type(data_object) == unicode:
node_type = "unicode"
else:
raise e
if node_type in self.lookup:
Mask = self.lookup[node_type]
return Mask(data_object)
return data_object
class NullMaskFactory(object):
def mask(self, data_object):
return data_object

View File

@ -1,40 +0,0 @@
class TargetHandler(object):
def __init__(self, project_info, node_factory, document):
self.project_info = project_info
self.node_factory = node_factory
self.document = document
def create_target(self, id_):
"""Creates a target node and registers it with the document and returns it in a list"""
target = self.node_factory.target(ids=[id_], names=[id_])
try:
self.document.note_explicit_target(target)
except Exception:
# TODO: We should really return a docutils warning node here
print("Warning: Duplicate target detected: %s" % id_)
return [target]
class NullTargetHandler(object):
def create_target(self, refid):
return []
class TargetHandlerFactory(object):
def __init__(self, node_factory):
self.node_factory = node_factory
def create_target_handler(self, options, project_info, document):
if options.has_key("no-link"):
return NullTargetHandler()
return TargetHandler(project_info, self.node_factory, document)

12
doc/source/conf.py → doc/conf.py Normal file → Executable file
View File

@ -33,7 +33,7 @@ extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.todo',
'sphinx.ext.coverage',
'sphinx.ext.pngmath',
'sphinx.ext.imgmath',
'sphinx.ext.ifconfig',
'sphinx.ext.viewcode',
'breathe',
@ -51,7 +51,7 @@ source_suffix = '.rst'
#source_encoding = 'utf-8-sig'
# The master toctree document.
master_doc = 'index'
master_doc = 'source/index'
# General information about the project.
project = u'OBITools3'
@ -292,7 +292,9 @@ texinfo_documents = [
#texinfo_no_detailmenu = False
#Breathe configuration
sys.path.append( "../breathe/" )
breathe_projects = { "OBITools3": "../doxygen/xml/" }
sys.path.append( "breathe/" )
breathe_projects = { "OBITools3": "doxygen/xml/" }
breathe_default_project = "OBITools3"
#breathe_projects_source = {
# "auto" : ( "../src", ["obidms.h", "obiavl.h"] )
# }

View File

@ -1,4 +0,0 @@
# Ignore everything in this directory
*
# Except this file
!.gitignore

94
doc/source/DMS.rst Normal file → Executable file
View File

@ -2,19 +2,19 @@
The OBItools3 Data Management System (OBIDMS)
*********************************************
A complete DNA Metabarcoding experiment rely on several kinds of data.
A complete DNA metabarcoding experiment relies on several kinds of data.
- The sequence data resulting of the PCR products sequencing,
- The sequence data resulting from the sequencing of the PCR products,
- The description of the samples including all their metadata,
- One or several refence database used for the taxonomical annotation
- One or several taxonomies.
- One or several reference databases used for the taxonomic annotation,
- One or several taxonomy databases.
Up to now each of these categories of data were stored in separate
files an nothing obliged to keep them together.
Up to now, each of these categories of data were stored in separate
files, and nothing made it mandatory to keep them together.
The `Data Management System` (DMS) of OBITools3 can be considered
as a basic database system.
The `Data Management System` (DMS) of OBITools3 can be viewed like a basic
database system.
OBIDMS UML
@ -25,11 +25,21 @@ OBIDMS UML
:download:`html version of the OBIDMS UML file <UML/ObiDMS_UML.class.violet.html>`
An OBIDMS directory consists of :
* OBIDMS column files
* OBIDMS release files
* OBIDMS dictionary files
* one OBIDMS history file
An OBIDMS directory contains :
* one `OBIDMS history file <#obidms-history-files>`_
* OBIDMS column directories
OBIDMS column directories
=========================
OBIDMS column directories contain :
* all the different versions of one OBIDMS column, under the form of different files (`OBIDMS column files <#obidms-column-files>`_)
* one `OBIDMS version file <#obidms-version-files>`_
The directory name is the column attribute with the extension ``.obicol``.
Example: ``count.obicol``
OBIDMS column files
@ -38,7 +48,7 @@ OBIDMS column files
Each OBIDMS column file contains :
* a header of a size equal to a multiple of PAGESIZE (PAGESIZE being equal to 4096 bytes
on most systems) containing metadata
* one column of data with the same OBIType
* Lines of data with the same `OBIType <types.html#obitypes>`_
Header
@ -48,27 +58,33 @@ The header of an OBIDMS column contains :
* Endian byte order
* Header size (PAGESIZE multiple)
*
* File status : Open/Closed
* Owner : PID of the process that created the file and is the only one allowed to modify it if it is open
* Number of lines (total or without the header?)
* OBIType
* Date of creation
* Version of the file
* Number of lines of data
* Number of lines of data used
* `OBIType <types.html#obitypes>`_ (type of the data)
* Date of creation of the file
* Version of the OBIDMS column
* The column name
* Eventual comments
Data
----
A column of data with the same OBIType.
A line of data corresponds to a vector of elements. Each element is associated with an element name.
Elements names are stored in the header. The correspondance between an element and its name is done
using their order in the lists of elements and elements names. This structure allows the storage of
dictionary-like data.
Example: In the header, the attribute ``elements_names`` will be associated with the value ``"sample_1;
sample_2;sample_3"``, and a line of data with the type ``OBInt_t`` will be stored as an ``OBInt_t`` vector
of size three e.g. ``5|8|4``.
Mandatory columns
-----------------
Some columns must exist in an OBIDMS directory :
* sequence identifiers column (type *OBIStr_t*)
* sequence identifiers column (type ``OBIStr_t``)
File name
@ -83,8 +99,7 @@ Example : ``count@3.odc``
Modifications
-------------
An OBIDMS column file can only be modified by the process that created it, if its status is set to Open. Those informations are
contained in the `header <#header>`_.
An OBIDMS column file can only be modified by the process that created it, and while its status is set to Open.
When a process wants to modify an OBIDMS column file that is closed, it must first clone it. Cloning creates a new version of the
file that belongs to the process, i.e., only that process can modify that file, as long as its status is set to Open. Once the process
@ -94,6 +109,8 @@ again.
That means that one column is stored in one file (if there is only one version)
or more (if there are several versions), and that there is one file per version.
All the versions of one column are stored in one directory.
Versioning
----------
@ -101,22 +118,22 @@ Versioning
The first version of a column file is numbered 0, and each new version increments that
number by 1.
The number of the latest version of an OBIDMS column is stored in an `OBIDMS release file <formats.html#obidms-release-files>`_.
The number of the latest version of an OBIDMS column is stored in the `OBIDMS version file <#obidms-version-files>`_ of its directory.
OBIDMS release files
OBIDMS version files
====================
Each OBIDMS column is associated with an OBIDMS release file that contains the number of the latest
Each OBIDMS column is associated with an OBIDMS version file in its directory, that contains the number of the latest
version of the column.
File name
---------
OBIDMS release files are named with the attribute associated to the data contained in the column, and
have the extension ``.odr``.
OBIDMS version files are named with the attribute associated to the data contained in the column, and
have the extension ``.odv``.
Example : ``count.odr``
Example : ``count.odv``
OBIDMS views
@ -140,19 +157,4 @@ operations ever done in the OBIDMS directory and the views in between them :
:width: 150 px
:align: center
OBIType header file
========================
.. doxygenfile:: obitypes.h
OBIIntColumn header file
========================
.. doxygenfile:: obiintcolumn.h
OBIColumn header file
=====================
.. doxygenfile:: obicolumn.h

BIN
doc/source/UML/OBIDMS_UML.png Normal file → Executable file

Binary file not shown.

Before

Width:  |  Height:  |  Size: 63 KiB

After

Width:  |  Height:  |  Size: 67 KiB

0
doc/source/UML/OBITypes_UML.class.violet.html Normal file → Executable file
View File

0
doc/source/UML/OBITypes_UML.png Normal file → Executable file
View File

Before

Width:  |  Height:  |  Size: 20 KiB

After

Width:  |  Height:  |  Size: 20 KiB

3162
doc/source/UML/ObiDMS_UML.class.violet.html Normal file → Executable file

File diff suppressed because it is too large Load Diff

14
doc/source/containers.rst Normal file → Executable file
View File

@ -2,28 +2,28 @@
Container types
===============
Containers allow to manage collection of values of homogeneous type.
Containers allow to manage collections of values of homogeneous type.
Three container types exist.
A container is a non-mutable structure once it has been locked.
Consequently just insert procedure are needed
Consequently, only insertion procedures are needed.
Lists
-----
Correspond to an ordered collection of values belonging an elementary type.
Correspond to an ordered collection of values belonging to an elementary type.
At its creation
At its creation, ...
Sets
----
Correspond to an unordered collection of values belonging an elementary type.
Correspond to an unordered collection of values belonging to an elementary type.
Dictionaries
------------
Dictionaries allow to associate a `key` to a `value`. Values can be retrieved through its associated key.
Values must belong an elementary type and keys must be *OBIStr_t*.
Dictionaries allow to associate a `key` to a `value`. Values can be retrieved through their associated key.
Values must belong to an elementary type and keys must be *OBIStr_t*.

4
doc/source/data.rst Normal file → Executable file
View File

@ -2,8 +2,8 @@
Data in OBITools3
#################
The OBITools3 inaugure a new way to manage DNA metabarcoding data.
They rely on a `Data management System` (DMS) that can be considered as
The OBITools3 introduce a new way to manage DNA metabarcoding data.
They rely on a `Data management System` (DMS) that can be viewed like
a simplified database system.

2
doc/source/elementary.rst Normal file → Executable file
View File

@ -12,7 +12,7 @@ Atomic types
========= ========= ============ ==============================
integer int32_t OBIInt_t a signed integer value
float double OBIFloat_t a floating value
boolean ? OBIBool_t a boolean true/false value
boolean bool OBIBool_t a boolean true/false value
char char OBIChar_t a character
index size_t OBIIdx_t an index in a data structure
========= ========= ============ ==============================

30
doc/source/guidelines.rst Normal file → Executable file
View File

@ -63,8 +63,14 @@ Issue tracking
==============
Issue tracking is done using `GitLab <https://about.gitlab.com/>`_ at http://git.metabarcoding.org/.
Creating a branch should always lead to the creation of a label that refers to it in GitLab.
Tickets should always be labelled with the branches for which they are relevant.
Tickets should always be labeled with the branches for which they are relevant.
*************
Documentation
*************
C functions are documented in the header files for public functions, and in the source file for private functions.
**************
@ -86,7 +92,7 @@ C99 :
* Object layer
* OBITools3 library
`Python 3 <https://www.python.org/>`_ :
`Python 3.5 <https://www.python.org/>`_ :
* Top layer code (scripts)
For the documentation, `Sphinx <http://sphinx-doc.org/>`_ should be used for both the original
@ -99,8 +105,22 @@ in the Sphinx documentation using `Breathe <https://breathe.readthedocs.org/en/l
Naming conventions
******************
.. todo::
Look for common naming conventions
Struct, Enum: ``Title_case``
Enum members, macros, constants: ``ALL_CAPS``
Functions, local variables: ``lower_case``
Public functions: ``obi_lower_case``
Functions that shouldn't be called directly: ``_lower_case`` (``_`` prefix)
Global variables: ``g_lower_case`` (``g_`` prefix)
Pointers: ``pointer_ptr`` (``_ptr`` suffix)
.. note::
Underscores are used to delimit 'words'.
*****************

0
doc/source/images/history.png Normal file → Executable file
View File

Before

Width:  |  Height:  |  Size: 17 KiB

After

Width:  |  Height:  |  Size: 17 KiB

0
doc/source/images/version_control.png Normal file → Executable file
View File

Before

Width:  |  Height:  |  Size: 48 KiB

After

Width:  |  Height:  |  Size: 48 KiB

Some files were not shown because too many files have changed in this diff Show More