obitools/obitools4
2
0
Fork 0
mirror of https://github.com/metabarcoding/obitools4.git synced 2026-08-08 06:20:36 +00:00
Code Issues Packages Projects Releases Wiki Activity
627 Commits 13 Branches 40 Tags
60f27c1dc87eb2cc799e4c6f963d9af9b6b8c70e
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Eric Coissac 60f27c1dc8 Add error handling for ambiguous bases in k-mer encoding 
This commit introduces error handling for ambiguous DNA bases (N, R, Y, W, S, K, M, B, D, H, V) in k-mer encoding. It adds new functions IterNormalizedKmersWithErrors and EncodeNormalizedKmersWithErrors that track and encode the number of ambiguous bases in each k-mer using error markers in the top 2 bits. The commit also updates the version string to reflect the latest changes.
 2026-02-04 21:45:08 +01:00
Eric Coissac 28162ac36f Ajout du filtre de fréquence avec v niveaux Roaring Bitmaps 
Implémentation complète du filtre de fréquence utilisant v niveaux de Roaring Bitmaps pour éliminer efficacement les erreurs de séquençage.

- Ajout de la logique de filtrage par fréquence avec v niveaux
- Intégration des bibliothèques RoaringBitmap et bitset
- Ajout d'exemples d'utilisation et de documentation
- Implémentation de l'itérateur de k-mers pour une utilisation mémoire efficace
- Optimisation pour les distributions skewed typiques du séquençage

Ce changement permet de filtrer les k-mers par fréquence minimale avec une utilisation mémoire optimale et une seule passe sur les données.
 2026-02-04 21:21:10 +01:00
Eric Coissac 1a1adb83ac Add error marker support for k-mers with enhanced documentation 
This commit introduces error marker functionality for k-mers with odd lengths up to 31. The top 2 bits of each k-mer are now reserved for error coding (0-3), allowing for error detection and correction capabilities. Key changes include:

- Added constants KmerErrorMask and KmerSequenceMask for bit manipulation
- Implemented SetKmerError, GetKmerError, and ClearKmerError functions
- Updated EncodeKmers, ExtractSuperKmers, EncodeNormalizedKmers functions to enforce k ≤ 31
- Enhanced ReverseComplement to preserve error bits during reverse complement operations
- Added comprehensive tests for error marker functionality including edge cases and integration tests

The maximum k-mer size is now capped at 31 to accommodate the error bits, ensuring that k-mers with odd lengths ≤ 31 utilize only 62 bits of the 64-bit uint64, leaving the top 2 bits available for error coding.
 2026-02-04 16:21:47 +01:00
Eric Coissac 05de9ca58e Add SuperKmer extraction functionality 
This commit introduces the ExtractSuperKmers function which identifies maximal subsequences where all consecutive k-mers share the same minimizer. It includes:

- SuperKmer struct to represent the maximal subsequences
- dequeItem struct for tracking minimizers in a sliding window
- Efficient algorithm using monotone deque for O(1) amortized minimizer tracking
- Comprehensive parameter validation
- Support for buffer reuse for performance optimization
- Extensive test cases covering basic functionality, edge cases, and performance benchmarks

The implementation uses simultaneous forward/reverse m-mer encoding for O(1) canonical m-mer computation and maintains a monotone deque to track minimizers efficiently.
 2026-02-04 16:04:06 +01:00
Eric Coissac 500144051a Add jj Makefile targets and k-mer encoding utilities 
Add new Makefile targets for jj operations (jjnew, jjpush, jjfetch) to streamline commit workflow.

Introduce k-mer encoding utilities in pkg/obikmer:
- EncodeKmers: converts DNA sequences to encoded k-mers
- ReverseComplement: computes reverse complement of k-mers
- NormalizeKmer: returns canonical form of k-mers
- EncodeNormalizedKmers: encodes sequences with normalized k-mers

Add comprehensive tests for k-mer encoding functions including edge cases, buffer reuse, and performance benchmarks.

Document k-mer index design for large genomes, covering:
- Use cases and objectives
- Volume estimations
- Distance metrics (Jaccard, Sørensen-Dice, Bray-Curtis)
- Indexing options (Bloom filters, sorted sets, MPHF)
- Optimization techniques (k-2-mer indexing)
- MinHash for distance acceleration
- Recommended architecture for presence/absence and counting queries
 2026-02-04 14:27:10 +01:00
Eric Coissac b49aba9c09 Implémentation du filtrage unique basé sur séquence et catégories 
Ajout d'une fonctionnalité pour le filtrage unique qui prend en compte à la fois la séquence et les catégories.

- Modification de la fonction ISequenceChunk pour accepter un classifieur unique optionnel
- Implémentation du traitement unique sur disque en utilisant un classifieur composite
- Mise à jour du classifieur utilisé pour le tri sur disque
- Correction de la gestion des clés de unicité en utilisant le code et la valeur du classifieur
- Mise à jour du numéro de commit
 2026-01-14 19:18:17 +01:00
Eric Coissac 0678181023 Refactor chunk processing and update version commit 
Optimize chunk processing by moving variable declarations inside the loop and update the commit hash in version.go to reflect the latest changes.
 2026-01-14 18:46:04 +01:00
Eric Coissac 371e702423 obiannotate --cut bug 2025-12-18 14:11:11 +01:00
Eric Coissac ac0d3f3fe4 Update obiuniq for very large dataset 2025-12-18 14:11:11 +01:00
Eric Coissac 547135c747 End of obilowmask 2025-12-03 11:49:07 +01:00
Eric Coissac e681666aaa End of obilowmask 2025-11-26 11:14:56 +01:00
Eric Coissac 272f5c9c35 End of obilowmask 2025-11-24 15:27:38 +01:00
Eric Coissac 86e60aedd0 obicsv bug with stat on value map fields 2025-11-21 14:03:31 +01:00
Eric Coissac 57c65f9d50 obimatrix bug 2025-11-21 13:24:24 +01:00
Eric Coissac e65b2a5efe obimatrix bugs 2025-11-21 13:24:06 +01:00
Eric Coissac ccc827afd3 finalise obilowmask 2025-11-18 15:33:08 +01:00
Eric Coissac cef29005a5 debug url reading 2025-11-18 15:30:20 +01:00
Eric Coissac 4603d7973e implementation de obilowmask 2025-11-18 15:30:20 +01:00
Eric Coissac 07cdd6f758 debug obimultiplex 
bug option obimultiplex
 2025-11-06 15:43:13 +01:00
Eric Coissac 2d7dc7d09d debug taxonomy core dump 2025-11-05 19:01:15 +01:00
Eric Coissac 7500ee1d15 update install script 2025-11-04 18:09:15 +01:00
Eric Coissac 0844dcc607 bug obimatrix 2025-10-28 13:57:31 +01:00
Eric Coissac 7f4ebe757e Bug obiuniq - don't clean the chunks 2025-10-28 13:50:22 +01:00
Eric Coissac d17a9520b9 work on obiclean chimera detection 2025-10-20 17:29:47 +02:00
Eric Coissac 29bf4ce871 add a feature to obimatrix adding obicsv option to obimatrix 2025-10-20 16:34:58 +02:00
Eric Coissac 82b6bb1ab6 correct a bug in func (worker SeqWorker) ChainWorkers(next SeqWorker) SeqWorker 2025-08-11 15:09:49 +02:00
Eric Coissac 6d204f6281 Patch the fastq detector 2025-08-08 10:23:03 -04:00
Eric Coissac 7a6d552450 Changes to be committed: 
modified:   pkg/obioptions/version.go
 2025-08-07 17:01:48 -04:00
Eric Coissac 412b54822c Patch a bug in obliclean for d>1 leading to some instability in the result 2025-08-07 17:01:38 -04:00
Eric Coissac 730d448fc3 Allows for only one cpu and it should work 2025-08-06 16:09:25 -04:00
Eric Coissac 04f3af3e60 some renaming of functions 2025-08-06 15:54:50 -04:00
Eric Coissac 997b6e8c01 correct the fastq detector for distinguish with a csv ngsfilter 2025-08-06 15:52:54 -04:00
Eric Coissac f239e8da92 Rename ISequenceChunk 2025-08-05 08:49:45 -04:00
Eric Coissac ed28d3fb5b Adds a --u-to-t option 2025-07-07 15:35:26 +02:00
Eric Coissac 43b285587e Debug on taxonomy extraction and CSV conversion 2025-07-07 15:29:40 +02:00
Eric Coissac 8d53d253d4 Add a reading option on readers to convet U to T 2025-07-07 15:29:07 +02:00
Eric Coissac 8c26fc9884 Add a new test on obisummary 2025-07-07 15:28:29 +02:00
Eric Coissac 235a7e202a Update obisummary to account new obiseq.StatsOnValues type 2025-06-19 17:21:30 +02:00
Eric Coissac 27fa984a63 Patch obimatrix accoring to the new type obiseq.StatsOnValues 2025-06-19 16:51:53 +02:00
Eric Coissac add9d89ccc Patch the Min and Max values of the expression language 2025-06-19 16:43:26 +02:00
Eric Coissac 9965370d85 Manage a lock on StatsOnValues 2025-06-17 16:46:11 +02:00
Eric Coissac 8a2bb1fe82 Changes to be committed: 
modified:   pkg/obioptions/version.go
	modified:   pkg/obiseq/merge.go
 2025-06-17 12:11:35 +02:00
Eric Coissac efc3f3af29 Patch a concurrent access problem 2025-06-17 12:05:42 +02:00
Eric Coissac 1c6ab1c559 Changes to be committed: 
modified:   pkg/obingslibrary/multimatch.go
	modified:   pkg/obioptions/version.go
 2025-06-17 09:06:42 +02:00
Eric Coissac 38dcd98d4a Patch the genbank parser automata 2025-06-17 08:52:45 +02:00
Eric Coissac 7b23985693 Add _ to allowed in taxid 2025-06-06 14:37:57 +02:00
Eric Coissac d31e677304 Patch a bug in obitag 2025-06-04 14:47:28 +02:00
Eric Coissac 6cb7a5a352 Changes to be committed: 
modified:   cmd/obitools/obitag/main.go
	modified:   cmd/obitools/obitaxonomy/main.go
	modified:   pkg/obiformats/csvtaxdump_read.go
	modified:   pkg/obiformats/ecopcr_read.go
	modified:   pkg/obiformats/ncbitaxdump_read.go
	modified:   pkg/obiformats/ncbitaxdump_readtar.go
	modified:   pkg/obiformats/newick_write.go
	modified:   pkg/obiformats/options.go
	modified:   pkg/obiformats/taxonomy_read.go
	modified:   pkg/obiformats/universal_read.go
	modified:   pkg/obiiter/extract_taxonomy.go
	modified:   pkg/obioptions/options.go
	modified:   pkg/obioptions/version.go
	new file:   pkg/obiphylo/tree.go
	modified:   pkg/obiseq/biosequenceslice.go
	modified:   pkg/obiseq/taxonomy_methods.go
	modified:   pkg/obitax/taxonomy.go
	modified:   pkg/obitax/taxonset.go
	modified:   pkg/obitools/obiconvert/sequence_reader.go
	modified:   pkg/obitools/obitag/obitag.go
	modified:   pkg/obitools/obitaxonomy/obitaxonomy.go
	modified:   pkg/obitools/obitaxonomy/options.go
	deleted:    sample/.DS_Store
 2025-06-04 09:48:10 +02:00
Eric Coissac 3424d3057f Changes to be committed: 
modified:   pkg/obiformats/ngsfilter_read.go
	modified:   pkg/obioptions/version.go
	modified:   pkg/obiutils/mimetypes.go
 2025-05-14 14:53:25 +02:00
Eric Coissac f9324dd8f4 add min and max to the obitools expression language 2025-05-13 16:03:03 +02:00
Eric Coissac f1b9ac4a13 Update the expression language 2025-05-07 20:45:05 +02:00
Eric Coissac e065e2963b Update the install script 2025-05-01 11:45:46 +02:00
Eric Coissac 13ff892ac9 Patch type mismatch in apat C library 2025-04-23 16:14:10 +02:00
Eric Coissac c0ecaf90ab Add the --number option to obiannotate 2025-04-22 18:35:51 +02:00
Eric Coissac a57cfda675 Make the replace function of the eval language accepting regex 2025-04-10 15:17:15 +02:00
Eric Coissac c2f38e737b Update of the packages 2025-04-10 15:16:36 +02:00
Eric Coissac 0aec5ba4df change the tests according to the corrections in obipairing 2025-04-04 17:10:17 +02:00
Eric Coissac 67e5b6ef24 Changes to be committed: 
modified:   pkg/obioptions/version.go
 2025-04-04 17:02:45 +02:00
Eric Coissac 3b1aa2869e Changes to be committed: 
modified:   pkg/obioptions/version.go
 2025-04-04 17:01:20 +02:00
Eric Coissac 7542e33010 Several bugs dicoverd during the doc writing 2025-04-04 16:59:27 +02:00
Eric Coissac 03b5ce9397 Patch a bug in obitag when some reference sequences have taxid absent from the taxonomy 2025-03-27 16:45:02 +01:00
Eric Coissac 2d52322876 Patch a bug in the obi2 annotation parser on map indexed by integers 2025-03-27 14:54:13 +01:00
Eric Coissac fd80249b85 Patch a bug in obitag when a taxon from the reference library is unknown in the taxonomy 2025-03-27 14:28:15 +01:00
Eric Coissac 5a3705b6bb Adds the --silent-warning options to the obitools commands and removes the --pared-with option from some of the obitols commands. 2025-03-25 16:44:46 +01:00
Eric Coissac 2ab6f67d58 Add a progress bar to chimera detection 2025-03-25 08:37:27 +01:00
Eric Coissac 8b379d30da Adds the --newick-output option to the obitaxonomy command 2025-03-14 14:24:12 +01:00
Eric Coissac 8448783499 Make sequence files recognized as a taxonomy 2025-03-14 14:22:22 +01:00
Eric Coissac d1c31c54de add a first version of the inline documentation 2025-03-12 14:40:42 +01:00
Eric Coissac 7a9dc1ab3b update release notes 2025-03-12 14:06:20 +01:00
Eric Coissac 3a1cf4fe97 Accelerate the speed of very long fasta sequences, and more generaly of every format 2025-03-12 13:29:41 +01:00
Eric Coissac 83926c91e1 Patch the install script to desactivate the CSV check 2025-03-12 13:28:52 +01:00
Eric Coissac 937a483aa6 Changes to be committed: 
modified:   Makefile
 2025-03-12 12:55:41 +01:00
Eric Coissac dada70e6b1 Changes to be committed: 
modified:   Makefile
 2025-03-12 12:49:34 +01:00
Eric Coissac 62e5a93492 update the compress option name 2025-03-11 17:14:40 +01:00
Eric Coissac f21f51ae62 Correct the logic of --update-taxid and --fail-on-taxonomy 2025-03-11 16:56:02 +01:00
Eric Coissac 3b5d4ba455 patch a bug in obiannotate 2025-03-11 16:35:38 +01:00
Eric Coissac 50d11ce374 Add a pre-push git-hook to run tests on obitools commands before pushing on master 2025-03-08 18:56:02 +01:00
Eric Coissac 52d5f6fe11 make makefile crashing on test error 2025-03-08 16:54:24 +01:00
Eric Coissac 78caabd2fd Add basic test on -h for all the commands 2025-03-08 16:28:06 +01:00
Eric Coissac 65bd29b955 normalize the usage of obitaxonomy 2025-03-08 13:00:55 +01:00
Eric Coissac b18c9b7ac6 add the --raw-taxid option 2025-03-08 09:40:06 +01:00
Eric Coissac 78df7db18d typos 2025-03-08 07:44:41 +01:00
Eric Coissac fc08c12ab0 update release notes 2025-03-08 07:42:20 +01:00
Eric Coissac 0339e4dffa Patch size limite of the filetype guesser 2025-03-08 07:34:02 +01:00
Eric Coissac 706b44c37f Add option for csv input format 2025-03-08 07:21:24 +01:00
Eric Coissac fbe7d15dc3 Changes to be committed: 
modified:   pkg/obioptions/version.go
	modified:   pkg/obitools/obicleandb/obicleandb.go
	modified:   pkg/obitools/obicleandb/options.go
 2025-03-06 13:38:38 +01:00
Eric Coissac b5cf586f17 patch a duplicate --taxonomy option in obirefidxdb 2025-03-06 11:36:20 +01:00
Eric Coissac 286e27d6ba patch the scienctific_name tag name to "scientific_name" 2025-03-05 14:22:12 +01:00
Eric Coissac 996ec69bd9 update the release notes for version 4.4.0 2025-03-01 12:56:39 +01:00
Eric Coissac 5f9182d25b Changes to be committed: 
modified:   pkg/obioptions/version.go
 2025-03-01 09:20:39 +01:00
Eric Coissac 9913fa8354 Changes to be committed: 
modified:   pkg/obioptions/version.go
 2025-03-01 09:14:56 +01:00
Eric Coissac 7b23314651 Some typos 2025-03-01 08:29:27 +01:00
Eric Coissac 1e541eac4c Last commit version 2025-03-01 08:24:26 +01:00
Eric Coissac 13cd4c86ac Patch the bug on --out with paired sequence files 2025-02-27 18:13:21 +01:00
Eric Coissac 75dd535201 Add a --valid-taxid option to obigrep 2025-02-27 18:12:55 +01:00
Eric Coissac 573acafafc Patch bug on ecotag with too short sequences 2025-02-27 15:09:07 +01:00
Eric Coissac 0067152c2b Patch the production of the ratio file 2025-02-27 10:19:39 +01:00
Eric Coissac 791d253edc Generate the ratio file as compressed if -Z option enabled. 2025-02-27 09:06:07 +01:00
Eric Coissac 6245d7f684 Changes to be committed: 
modified:   .gitignore
 2025-02-24 15:47:45 +01:00
Eric Coissac 13d610aff7 Changes to be committed: 
modified:   pkg/obioptions/version.go
	modified:   pkg/obitools/obiclean/chimera.go
 2025-02-24 15:25:45 +01:00