1e1f575d1c
refactor: replace single batch size with min/max bounds and memory limits
...
Introduce separate _BatchSize (min) and _BatchSizeMax (max) constants to replace the single _BatchSize variable. Update RebatchBySize to accept both maxBytes and maxCount parameters, flushing when either limit is exceeded. Set default batch size min to 1, max to 2000, and memory limit to 128 MB. Update CLI options and sequence_reader.go accordingly.
2026-03-13 15:07:35 +01:00
40769bf827
Add memory-based batching support
...
Implement memory-aware batch sizing with --batch-mem CLI option, enabling adaptive batching based on estimated sequence memory footprint. Key changes:
- Added _BatchMem and related getters/setters in pkg/obidefault
- Implemented RebatchBySize() in pkg/obiter for memory-constrained batching
- Added BioSequence.MemorySize() for conservative memory estimation
- Integrated batch-mem option in pkg/obioptions with human-readable size parsing (e.g., 128K, 64M, 1G)
- Added obiutils.ParseMemSize/FormatMemSize for unit conversion
- Enhanced pool GC in pkg/obiseq/pool.go to trigger explicit GC for large slice discards
- Updated sequence_reader.go to apply memory-based rebatching when enabled
2026-03-13 14:54:21 +01:00
1ce5da9bee
Support new sequence file formats and improve error handling
...
Add support for .gbff and .gbff.gz file extensions in sequence reader.
Update the logic to return an error instead of using NilIBioSequence when no sequence files are found, improving the error handling and user feedback.
2026-02-11 06:31:10 +01:00
1a28d5ed64
Add progress bar configuration and conditional display
...
This commit introduces a new configuration module `obidefault` to manage progress bar settings, allowing users to disable progress bars via a `--no-progressbar` option. It updates various packages to conditionally display progress bars based on this new configuration, improving user experience by providing control over progress bar output. The changes also include improvements to progress bar handling in several packages, ensuring they are only displayed when appropriate (e.g., when stderr is a terminal and stdout is not piped).
2026-02-08 16:14:02 +01:00
7c12b1ee83
Disable progress bar when output is piped
...
Modify CLIProgressBar function to check if stdout is a named pipe and disable the progress bar accordingly. This prevents the progress bar from being displayed when the output is redirected or piped to another command.
2026-02-08 14:48:13 +01:00
cef29005a5
debug url reading
2025-11-18 15:30:20 +01:00
ed28d3fb5b
Adds a --u-to-t option
2025-07-07 15:35:26 +02:00
6cb7a5a352
Changes to be committed:
...
modified: cmd/obitools/obitag/main.go
modified: cmd/obitools/obitaxonomy/main.go
modified: pkg/obiformats/csvtaxdump_read.go
modified: pkg/obiformats/ecopcr_read.go
modified: pkg/obiformats/ncbitaxdump_read.go
modified: pkg/obiformats/ncbitaxdump_readtar.go
modified: pkg/obiformats/newick_write.go
modified: pkg/obiformats/options.go
modified: pkg/obiformats/taxonomy_read.go
modified: pkg/obiformats/universal_read.go
modified: pkg/obiiter/extract_taxonomy.go
modified: pkg/obioptions/options.go
modified: pkg/obioptions/version.go
new file: pkg/obiphylo/tree.go
modified: pkg/obiseq/biosequenceslice.go
modified: pkg/obiseq/taxonomy_methods.go
modified: pkg/obitax/taxonomy.go
modified: pkg/obitax/taxonset.go
modified: pkg/obitools/obiconvert/sequence_reader.go
modified: pkg/obitools/obitag/obitag.go
modified: pkg/obitools/obitaxonomy/obitaxonomy.go
modified: pkg/obitools/obitaxonomy/options.go
deleted: sample/.DS_Store
2025-06-04 09:48:10 +02:00
5a3705b6bb
Adds the --silent-warning options to the obitools commands and removes the --pared-with option from some of the obitols commands.
2025-03-25 16:44:46 +01:00
2ab6f67d58
Add a progress bar to chimera detection
2025-03-25 08:37:27 +01:00
62e5a93492
update the compress option name
2025-03-11 17:14:40 +01:00
706b44c37f
Add option for csv input format
2025-03-08 07:21:24 +01:00
7b23314651
Some typos
2025-03-01 08:29:27 +01:00
13cd4c86ac
Patch the bug on --out with paired sequence files
2025-02-27 18:13:21 +01:00
773e54965d
Patch a bug on compressed output
2025-02-05 14:18:24 +01:00
ceca33998b
add extensions fq in directory scanning
2025-02-04 20:34:58 +01:00
2452aef7a9
patch multiple -Z options
2025-01-29 21:35:28 +01:00
337954592d
add the --out option to the obitaxonomy
2025-01-29 13:22:35 +01:00
7c4042df6b
introduce obidefault
2025-01-27 17:12:45 +01:00
9acb4a85a8
Refactoring of the default values
2025-01-24 18:09:59 +01:00
ccd3b06532
Merge branch 'master' into taxonomy
2024-12-20 20:06:57 +01:00
5d0f996625
Patch a small bug on json write
2024-12-20 19:42:03 +01:00
40fb4e9767
reduce the memory impact of obiuniq.
2024-11-27 13:30:16 +01:00
05bf2bfd6c
Add option related to agrep match on obigrep and obiannotate
2024-09-09 16:52:13 +02:00
31bfc88eb9
Patch a bug on writing to stdout, and add clearer error on openning data files
2024-08-13 09:45:28 +02:00
1b1cd41fd3
Add some code refactoring from the blackboard branch
2024-08-02 12:35:46 +02:00
4e4fac491f
Fisrt versin of the two levels indexing
...
Former-commit-id: 4d86483bc120e27cb6f5d2c216596d410274fc69
2024-07-12 15:17:48 +02:00
e6b87ecd02
Reduce memory allocation events
...
Former-commit-id: fbdb2afc857b02adc2593e2278d3bd838e99b0b2
2024-06-22 21:01:53 +02:00
98b3bc2a8c
Patch a bug on the reading of each last sequence of a chunck in the fasta reader
...
Former-commit-id: eacf64112582befa4751f66352999a28abf349f7
2024-05-27 10:17:17 +02:00
55ce36f329
Update of obipcr and homogenization of logging
...
Former-commit-id: 46abf47c19ace5248042c02cf1f81d9f6c12eb10
2024-05-16 15:18:30 +02:00
d30d736e48
Simplify the workers code by removing dupplicates
...
Former-commit-id: 638fcf8d88dd93755d1ec89c8fe92f6ed3f733df
2024-04-30 12:22:22 +02:00
f37477b26c
Progress bar are automagically disabled if stderr is redirected
...
Former-commit-id: 20a3a22db7dcf530b9e9cad9090b46f22db0d078
2024-04-22 12:23:59 +02:00
coissac
8d77cc4133
Change path of the obitools pkg
...
Former-commit-id: 311cbf8df3b990b393c6f4885d62e74564423b65
2023-11-29 12:14:37 +01:00
coissac
185b974d13
Adds a JSON output format
...
Former-commit-id: 26f07460772c0f735bf705d473f892878d3e57f0
2023-11-07 11:56:49 +02:00
coissac
61c30f9b6a
Patch rev complement and first implementation of --auto in obicsv
...
Former-commit-id: f3020e81283b1073c4d1c2d2ff0887e3998e6764
2023-11-07 09:37:07 +02:00
coissac
be47ec909c
add the --skip-empty option
...
Former-commit-id: ec9cb0ecaf90a61bf9289cf4c089b5cc2fcb65a5
2023-07-17 14:24:02 +02:00
coissac
d70bb45f3f
Small change in parallelisation tuning
...
Former-commit-id: 3fe2495b7fd86a0ba47dd87907323a457bae481a
2023-04-05 14:15:47 +02:00
coissac
a33e471b39
First attempt for obiconsensus... The graph traversing algorithm is too simple
...
Former-commit-id: 0456e6c7fd55d6d0fcf9856c40386b976b912cba
2023-03-27 19:51:10 +07:00
coissac
5fbe52368c
Patch the empty batch bug
...
Former-commit-id: fcee04b58f2c4a0bf2c27792f991391c0b6ce78e
2023-03-07 20:16:06 +07:00
coissac
b3922c3896
Produce less weird crash on non existing files
...
Former-commit-id: 74bb27bd53c685be530632994bd2ba24c1f362e1
2023-03-07 17:34:25 +07:00
coissac
d88de15cdc
Refactoring codes for removing buffer size options. An some other changes...
...
Former-commit-id: 10b57cc1a27446ade3c444217341e9651e89cdce
2023-03-07 11:12:13 +07:00
coissac
072b85e155
change the model for representing paired reads and extend its usage to other commands
2023-02-23 23:35:58 +01:00
coissac
56722895e4
Adds to every obitools the ability to save in gzip compressed files
2023-02-18 12:06:52 +01:00
coissac
9554a32490
Add capacity to obidistribute to save gzipped files
2023-02-17 12:52:19 +01:00
coissac
6e36b22040
Normalise the managment of speed measure
2023-02-16 13:31:49 +01:00
coissac
f97f92df72
rename the iterator class
2023-01-22 22:04:17 +01:00
coissac
09fc426b67
Refactoring related to iterators
2022-11-16 17:13:03 +01:00
coissac
6f853da9df
Remove single sequence ierators. Only batch iterators persist
2022-11-16 10:58:59 +01:00
coissac
65210755cc
Patch reading of annotation from stdin
2022-09-29 14:48:05 +02:00
coissac
3cd8aabd83
Correct some misspelling in option helps
...
Changes to be committed:
modified: pkg/obitools/obiclean/options.go
modified: pkg/obitools/obiconvert/options.go
2022-09-20 21:59:38 +02:00
Eric Coissac
coissac