Eric Coissac
40769bf827
Add memory-based batching support
...
Implement memory-aware batch sizing with --batch-mem CLI option, enabling adaptive batching based on estimated sequence memory footprint. Key changes:
- Added _BatchMem and related getters/setters in pkg/obidefault
- Implemented RebatchBySize() in pkg/obiter for memory-constrained batching
- Added BioSequence.MemorySize() for conservative memory estimation
- Integrated batch-mem option in pkg/obioptions with human-readable size parsing (e.g., 128K, 64M, 1G)
- Added obiutils.ParseMemSize/FormatMemSize for unit conversion
- Enhanced pool GC in pkg/obiseq/pool.go to trigger explicit GC for large slice discards
- Updated sequence_reader.go to apply memory-based rebatching when enabled
2026-03-13 14:54:21 +01:00
Eric Coissac
6ee8750635
Replace SplitInTwo with LeftSplitInTwo/RightSplitInTwo for precise splitting
...
Replace SplitInTwo calls with LeftSplitInTwo or RightSplitInTwo depending on the intended split direction. In fastseq_json_header.go, extract rank from suffix without splitting; in biosequenceslice.go and taxid.go, use LeftSplitInTwo to split from the left; add RightSplitInTwo utility function for splitting from the right.
2026-03-12 18:41:28 +01:00
Eric Coissac
4603d7973e
implementation de obilowmask
2025-11-18 15:30:20 +01:00
Eric Coissac
6d204f6281
Patch the fastq detector
2025-08-08 10:23:03 -04:00
Eric Coissac
997b6e8c01
correct the fastq detector for distinguish with a csv ngsfilter
2025-08-06 15:52:54 -04:00
Eric Coissac
235a7e202a
Update obisummary to account new obiseq.StatsOnValues type
2025-06-19 17:21:30 +02:00
Eric Coissac
add9d89ccc
Patch the Min and Max values of the expression language
2025-06-19 16:43:26 +02:00
Eric Coissac
7b23985693
Add _ to allowed in taxid
2025-06-06 14:37:57 +02:00
Eric Coissac
3424d3057f
Changes to be committed:
...
modified: pkg/obiformats/ngsfilter_read.go
modified: pkg/obioptions/version.go
modified: pkg/obiutils/mimetypes.go
2025-05-14 14:53:25 +02:00
Eric Coissac
f9324dd8f4
add min and max to the obitools expression language
2025-05-13 16:03:03 +02:00
Eric Coissac
f1b9ac4a13
Update the expression language
2025-05-07 20:45:05 +02:00
Eric Coissac
a57cfda675
Make the replace function of the eval language accepting regex
2025-04-10 15:17:15 +02:00
Eric Coissac
8448783499
Make sequence files recognized as a taxonomy
2025-03-14 14:22:22 +01:00
Eric Coissac
0067152c2b
Patch the production of the ratio file
2025-02-27 10:19:39 +01:00
Eric Coissac
0df082da06
Adds possibility to extract a taxonomy from taxonomic path included in sequence files
2025-01-30 11:18:21 +01:00
Eric Coissac
8a28c9ae7c
add the --download-ncbi option to obitaxonomy
2025-01-29 12:38:39 +01:00
Eric Coissac
9acb4a85a8
Refactoring of the default values
2025-01-24 18:09:59 +01:00
Eric Coissac
3137c1f841
Adds the ability to read gzip-tar file for the taxonomy dump
2025-01-24 11:47:59 +01:00
Eric Coissac
5d0f996625
Patch a small bug on json write
2024-12-20 19:42:03 +01:00
Eric Coissac
795df34d1a
Changes to be committed:
...
modified: cmd/obitools/obitag/main.go
modified: cmd/obitools/obitag2/main.go
modified: go.mod
modified: go.sum
modified: pkg/obiformats/ncbitaxdump/read.go
modified: pkg/obioptions/version.go
modified: pkg/obiseq/attributes.go
modified: pkg/obiseq/taxonomy_lca.go
modified: pkg/obiseq/taxonomy_methods.go
modified: pkg/obiseq/taxonomy_predicate.go
modified: pkg/obitax/inner.go
modified: pkg/obitax/lca.go
new file: pkg/obitax/taxid.go
modified: pkg/obitax/taxon.go
modified: pkg/obitax/taxonomy.go
modified: pkg/obitax/taxonslice.go
modified: pkg/obitools/obicleandb/obicleandb.go
modified: pkg/obitools/obigrep/options.go
modified: pkg/obitools/obilandmark/obilandmark.go
modified: pkg/obitools/obilandmark/options.go
modified: pkg/obitools/obirefidx/famlilyindexing.go
modified: pkg/obitools/obirefidx/geomindexing.go
modified: pkg/obitools/obirefidx/obirefidx.go
modified: pkg/obitools/obirefidx/options.go
modified: pkg/obitools/obitag/obigeomtag.go
modified: pkg/obitools/obitag/obitag.go
modified: pkg/obitools/obitag/options.go
modified: pkg/obiutils/strings.go
2024-12-19 13:36:59 +01:00
Eric Coissac
9471fedfa1
Fisrt step in the obitax rewriting
2024-11-08 09:48:16 +01:00
Eric Coissac
fefc360f80
Changes to be committed:
...
modified: pkg/obioptions/version.go
modified: pkg/obitools/obitagpcr/pcrtag.go
modified: pkg/obiutils/abs.go
new file: pkg/obiutils/abs_test.go
2024-10-28 21:51:21 +01:00
Eric Coissac
31bfc88eb9
Patch a bug on writing to stdout, and add clearer error on openning data files
2024-08-13 09:45:28 +02:00
Eric Coissac
bdb96dda94
Adds the obimicrosat command
2024-08-05 15:31:20 +02:00
Eric Coissac
67665a6b40
Xprize update
...
Former-commit-id: d38919a897961e4d40da3b844057c3fb94fdb6d7
2024-07-25 18:09:03 -04:00
Eric Coissac
4e4fac491f
Fisrt versin of the two levels indexing
...
Former-commit-id: 4d86483bc120e27cb6f5d2c216596d410274fc69
2024-07-12 15:17:48 +02:00
Eric Coissac
bd855c4965
Adds CSV as an input format
...
Former-commit-id: a365bb6947064adc2709d66df05fa54c6fe47fad
2024-07-03 21:04:27 +02:00
Eric Coissac
1835cb2cf3
remove sonic which only works on AMD64
...
Former-commit-id: 5073957b4706b68896006ee0f7bc30e49ff72c3b
2024-06-24 14:53:18 +02:00
Eric Coissac
c1f03cb1f6
Switch to faster json library go-json and sonic
...
Former-commit-id: ab9b4723f1dcf79fe5c073fff4d86f4f6969edfd
2024-06-23 00:36:08 +02:00
Eric Coissac
93f9dcb95f
Reducing memory allocation events
...
Former-commit-id: c94e79ba116464504580fc397270ead154063971
2024-06-22 22:32:31 +02:00
Eric Coissac
a812fe8b72
debug of obimultiplex
...
Former-commit-id: 1cf276840feb9d6135d96bd1bf63627d3085ae6e
2024-06-18 23:38:33 +02:00
Eric Coissac
65f5109957
Plenty of small bugs
...
Former-commit-id: 42c7fab7d65906c80ab4cd32da6867ff21842ea8
2024-06-04 16:49:12 +02:00
Eric Coissac
dd9307a4cd
Swich to the system min and max functions and remove the version from obiutils
...
Former-commit-id: 8c4558921b0d0c266b070f16e83813de6e6d4a0f
2024-05-30 08:27:24 +02:00
Eric Coissac
55ce36f329
Update of obipcr and homogenization of logging
...
Former-commit-id: 46abf47c19ace5248042c02cf1f81d9f6c12eb10
2024-05-16 15:18:30 +02:00
Eric Coissac
017030bcce
Add obiminion first version
...
Former-commit-id: aa5ace7bd4d2266333715fca7094d1c3cbbb5e6d
2024-05-14 08:16:12 +02:00
2caaa62485
Implements the kmeans++ algo to select the landmarks in the geometric method
...
Former-commit-id: 732404a0dc6d7276e4e479dd2481aa4bd42d4ce5
2023-12-11 16:07:03 +01:00
8d77cc4133
Change path of the obitools pkg
...
Former-commit-id: 311cbf8df3b990b393c6f4885d62e74564423b65
2023-11-29 12:14:37 +01:00
8620ea1637
First version of obimatrix
...
Former-commit-id: 6e09eb0dd75bc688a6c83ef40dd88658fb1b296e
2023-11-12 20:40:56 +01:00
5ea2b8afcf
a first version of obisummary
...
Former-commit-id: cca1019d82a14a322f46a20890b996b5c7491d41
2023-11-09 22:33:06 +02:00
d23a911080
Change the way sequence definition are managed. They are now when present stored as an attribute
...
Former-commit-id: 6e618377c05b42937d2eace3c9668390980ab68c
2023-10-05 07:21:12 +02:00
3f8c0d6a2f
Replace MakeBioSequence call by NewBioSequence call,
...
Implements a new file format guesser
Adds some more API doc
Former-commit-id: 9837bf1c28beca6ddb599b367f93548950ba83c1
2023-08-30 19:59:46 +02:00
0303244e3c
Adds doc and unit tests
...
Former-commit-id: 6f60f97947111b4fc65cc0e72d18770ba6094088
2023-08-27 16:48:15 +02:00
cd7414d471
Unit tests for obiutils.Set
...
Former-commit-id: 352ec0da26184d13f74a34a1ad70ca4edef7e4a6
2023-08-27 16:40:41 +02:00
bfe6fa630a
Adds unit tests
...
Former-commit-id: 33e9d9a8a370afc3d6318c6972de02a957fa3098
2023-08-27 16:23:09 +02:00
5b7d88ccc6
Patch a bug in the Dim function
...
Former-commit-id: 004545e81df4dcf33294fb062de8204692035cdf
2023-08-27 16:18:10 +02:00
7b7128196a
small refactoring including doc
...
Former-commit-id: 6d6b527d89d77aa571831f7500f841840e280536
2023-08-27 15:52:28 +02:00
9bf006af93
A first prototype for the space of sequences
...
Former-commit-id: 07dc6ef044b5b6a6fb45dc2acb01dffe71a96195
2023-08-27 14:58:55 +02:00
077f3b5bb5
first trial of obilandmark
...
Former-commit-id: 00a50bdbf407b03dfdc385a848a536559f5966a5
2023-08-25 23:23:23 +02:00
2a11adb346
Add some doc and switch to the parallel gzip library
...
Former-commit-id: 2c1187001f989ba3de5895f516d4c8b54d52a4c4
2023-08-25 14:36:38 +02:00
3778ae9266
Correction a a bug and transfert of the new matching rules from obirefidx to obitag
...
Former-commit-id: a28540f72a504ad4e7e8a8b6f7609116913445b4
2023-05-05 07:37:19 +02:00
