Corrects duplicated typos in the registered GVal function names, changing "which_maxwhichmax" and "which_minwhichmin" to "which_max" and "which_min". The underlying obiutils.WhichMax/WhichMin logic and its int-to-float64 index conversion remain unchanged.
Implement reflection-based WhichMax and WhichMin to dynamically find the index or key of the maximum/minimum element in slices, arrays, or maps. Functions validate orderability, handle empty collections, and dispatch via reflect.Kind. Expose as which_max and which_min GVal functions, with float64 type assertions for compatibility and preserved error handling.
Introduces an `unwrapInterface` reflection helper to dereference `interface{}`-wrapped values before type validation. Updates slice and map iteration loops in min/max functions to apply this helper, ensuring `isOrderedKind` accurately identifies underlying concrete types instead of incorrectly rejecting `reflect.Interface` elements.
Extends OBI header parsing to recognize and deserialize JSON-like arrays and objects. Introduces safe conversion utilities in `obiutils` to cast generic interface values into typed maps, and exposes them via new `BioSequence` methods. Header values are now marshaled, quote-normalized, and formatted for map and slice types.
Multiple raw k-mers collapsing into identical circular canonical forms introduce bias into complexity estimates. This change pre-computes `log(class_size)` tables and per-word-size maximum entropy bounds. The `KmerEntropy` function and `KmerEntropyFilter` are updated to apply the corrected formula `(log(N) + Σf·log(s) - Σf·log(f))/N / emax`, ensuring accurate sequence complexity estimation.
Introduce generic and reflection-based utilities for filtering slices and maps by minimum/maximum thresholds, along with saturating subtraction. The `obiutils` package provides type-safe generic implementations alongside dynamic reflection dispatchers to handle arbitrary ordered and numeric types. These are exposed as GVAL expression functions in `obiseq`, extending the language's built-in filtering and numeric capabilities.
Adds a pre-processing guard that checks for empty sequence identifiers before formatting. This prevents malformed FASTA output and stops downstream processing of invalid FASTQ data by terminating early. The check is placed before existing sequence-length validations to enforce non-empty IDs during batch processing.
- **obiconvert**: Added `--raw-taxid` mode to output numeric taxIDs without formatting (e.g., "12345" instead of ":tax:NCBI_0987@species"). Introduced `TaxNode.FullString()` to reliably return full formatted strings regardless of global settings, and improved fallback behavior when taxonomy DB is unavailable.
- **ngsfilter**: Input fields (primers, sample tags/IDs) are now automatically trimmed of leading/trailing whitespace to prevent parsing failures from inconsistent formatting.
- **obitools (pcrtag)**: Mismatch-related fields (`forward_mismatches`, `reverse_mishaps`) renamed to "error" for consistency across annotation dictionaries.
- **obipairing & obtagpcr**: Enforced mandatory paired-end file input (`--forward` and `reverse`) in obipairing; added CLI support for generating config templates via AskConfigTemplate(); removed redundant `Required()` constraints and introduced helper function CLIHasPairedFiles().
- Add new `--tax-id` mode (`obiconvert --raw-taxid`) to output bare numeric taxIDs instead of full-format strings.
- Introduce `TaxNode.FullString()` to always return the complete "code:id [name]@rank" format, regardless of global `UseRawTaxids()` setting.
- Update `.String(taxonomyCode)` to respect the global flag, returning bare ID when `--raw-taxid` is active.
- Extract raw taxID from full-format strings in taxonomy methods when needed (e.g., fallback without loaded DB).
- Add comprehensive test suite covering:
a) `--raw-taxid` execution and idempotency
b) full-format taxID output with `--taxonomy`
c interaction of both flags
d format validation
- Add test data: new reference files `out_ecotag.fasta`, taxonomy.csv, and updated shell script.
Trim leading/trailing whitespaces in forward/reverse primers, tags (via sample_tag), experiment andsample fields to prevent parsing errors due to formatting inconsistencies in input data.
- Renamed `obimultiplex_forward_mismatches` to "error" for consistency
- Similarly renamed
`obimultiplex_reverse_mismatches` to "error"
- Applied changes in both annotation dictionaries (aanot, banot)
- Enforce requirement of both forward (-F) and reverse files in obipairing/main.go
- Add config template support to obtagpcr via CLIAskConfigTemplate()
- Remove redundant Required() constraints in options.go
- Introduce new helper CLIHasPairedFiles()
- Added strict length matching between sequences and quality scores in `SetQualities`, `Take Qualites` (note: likely intended as " TakeQuantiles" or similar, but preserved per commit), and `Subsequence` operations; an error is now raised if lengths do not match.
- Introduced a new `map_summaries` aggregation feature in obisummary to merge map summary data across datasets, supporting safe concurrent access and inclusion of non-empty results in the final output.
- Centralized string reversal logic via a new `inverser_chaine()` utility function, replacing duplicated inline implementations throughout the codebase.
- Implement merging logic of `map summaries` across datasets
- Ensure proper initialization and population in multi-threaded context
- Add `map_summaries` to final output dictionary when non-empty
[obiseq] Add length validation for qualities in SetQualities, Take Qualites and Subsequence
- Panic if sequence/qualities length mismatch when setting or taking qualities in BioSequence.
- Add same check before slicing Qualities() for Subsequence to ensure consistency.
Refactor pushInterfaceToLua to delegate unsupported types (nil, bool/int/float/string/map/slice) recursively via new lvalueFromInterface helper. Simplify typed slice and map handlers, remove explicit nil case (now handled by lvalueFromInterface), eliminate redundant type switches in pushMapStringIntToLua and similar functions. Add new luajson.go with RegisterJSON, lua.JSONEncode/Decode bindings using lvalueFromInterface and Table2 Interface for bidirectional round-trips. Include comprehensive tests covering scalars, nested structures (e.g., kmindex response), arrays and error cases.
- **Bug fix**: Corrected logic in 4-mer calculation to properly handle sequences of length exactly three. Previously, such cases could produce invalid or unexpected results due to an incomplete guard condition (`length < 0`) which failed for ` length == 3` (where computed step size was zero). The fix ensures all sequences shorter than four bases are safely excluded.
- **Refactor**: Introduced a new internal utility function (`inverser_chaine`) to centralize string reversal logic, improving code maintainability and test coverage without affecting user-facing behavior.
- Improved concurrency safety by replacing the global HTTP client with a thread-safe, lazy-initialized instance using `sync.Once`. The new implementation enables connection pooling (`MaxIdleConnsPerHost`, connections per host) and dynamically configures pool size based on `obidefault.ParallelWorkers()`, ensuring robust behavior in multi-threaded Lua environments.
- Updated GitHub Actions workflows to the latest stable versions of `actions/setup-go` and ` actions/checkout`, improving build reliability.
- Removed outdated Go dependency checksums for buger/jsonparser v1.1.x to keep the build clean and consistent.
- Replace global _httpClient variable by a sync.Once-based lazy initialization
- Add getHTTPClient() function to safely initialize client with connection pooling settings (MaxIdleConnsPerHost, Max Con ns/Conn per host)
- Set connection pool size based on obidefault.ParallelWorkers()
This ensures safe concurrent access and better resource management in multi-threaded Lua environments.
//
// Registers the http module in Lua state as a global,
// aligning with obicontext and BioSequence conventions.
The change ensures consistent module exposure across Lua environments.
- Update obioptions/version.go and version.txt from Release v4.5 to 68302a1
- Increment patch version: from `Release v4.5` → 68302a1
- Align version.txt with current release tag
In the 4mer calculation:
length := slength - 3
- for sequences with <4 bases, length is <=0
The check to stop did only catch <0, so sequences lengths 2 or less, leaving sequence lengths of 3 unguarded
if length < 0 {
return nil
}
This release introduces dynamic batch flushing in the Distribute component, replacing the previous fixed-size batching with a memory- and count-aware strategy. Batches now flush automatically when either the maximum sequence count (BatchSizeMax()) or memory threshold (BatchMem()) per key is reached, ensuring more efficient resource usage and consistent behavior with the RebatchBySize strategy. The optional sizes parameter has been removed, and related code—including the Lua wrapper and worker buffer handling—has been updated for correctness and simplicity. Unused BatchSize() references have been eliminated from obidistribute.
Additionally, this release includes improvements to static Linux builds and overall build stability, enhancing reliability across deployment environments.
Replace the old fixed batch-size mechanism in Distribute with a dynamic strategy that flushes batches when either BatchSizeMax() sequences or BatchMem() bytes are reached per key. This aligns with the RebatchBySize strategy and removes the optional sizes parameter. Also update related code: simplify Lua wrapper to accept optional capacity, and fix buffer growth logic in worker.go using slices.Grow correctly. Remove unused BatchSize() usage from obidistribute.
This release focuses on improving build reliability, memory efficiency for large datasets, and portability of Linux binaries.
### Static Linux Binaries
- Linux binaries are now built with static linking using musl, eliminating external runtime dependencies and ensuring portability across distributions.
### Memory-Aware Batching
- Users can now control memory usage during processing with the new `--batch-mem` option, specifying limits such as 128K, 64M, or 1G.
- Batching logic now respects both size and memory constraints: batches are flushed when either threshold is exceeded.
- Conservative memory estimation for sequences helps avoid over-allocation, and explicit garbage collection after large batch discards reduces memory spikes.
### Build System Improvements
- Upgraded to Go 1.26 for improved performance and toolchain stability.
- Fixed cross-compilation issues by replacing generic include paths with architecture-specific ones (x86_64-linux-gnu and aarch64-linux-gnu).
- Streamlined macOS builds by removing special flags, using standard `make` targets.
- Enhanced error reporting during build failures: logs are now shown before cleanup and exit.
- Updated install script to correctly configure GOROOT, GOPATH, and GOTOOLCHAIN, with visual progress feedback for downloads.
All batching behavior is non-breaking and maintains backward compatibility while offering more predictable resource usage on large datasets.
Update version to 4.4.27 in version.txt and pkg/obioptions/version.go.
Add zlib-static package to release workflow to ensure static linking of zlib, resolving potential runtime dependency issues with the external link mode.
- Upgrade Go version from 1.23 to 1.26 in release.yml
- Remove CGO_CFLAGS from cross-compilation matrix entries
- Replace Linux build tools installation with Docker-based static build using golang:1.26-alpine
- Simplify macOS build to use standard make without special flags
- Increment version to 4.4.26
Update version to 4.4.25 in version.txt and pkg/obioptions/version.go.
Fix CGO_CFLAGS in release.yml by replacing generic '-I/usr/include' with architecture-specific paths (x86_64-linux-gnu and aarch64-linux-gnu) to ensure correct header inclusion during cross-compilation on Linux.
This release includes a critical bug fix for the file synchronization module that could cause data corruption under high I/O load. Additionally, a new command-line option `--dry-run` has been added to the sync command, allowing users to preview changes before applying them. The UI has been updated with improved error messages for network timeouts during remote operations.
- Update version from 4.4.22 to 4.4.23 in version.txt and pkg/obioptions/version.go
- Add zlib1g-dev dependency to Linux release workflow for potential linking requirements
- Improve tag creation in Makefile by resolving commit hash with `jj log` for better CI/CD integration
### Memory-Aware Batching
- Replaced single batch size limits with configurable min/max bounds and memory limits for more precise control over resource usage.
- Added `--batch-mem` CLI option to enable adaptive batching based on estimated sequence memory footprint (e.g., 128K, 64M, 1G).
- Introduced `RebatchBySize()` with explicit support for both byte and count limits, flushing when either threshold is exceeded.
- Implemented conservative memory estimation via `BioSequence.MemorySize()` and enhanced garbage collection to trigger explicit cleanup after large batch discards.
- Updated internal batching logic across `batchiterator.go`, `fragment.go`, and `obirefidx.go` to consistently use default memory (128 MB) and size (min: 1, max: 2000) bounds.
### Linux Build Enhancements
- Enabled static linking for Linux binaries using musl, producing portable, self-contained executables without external dependencies.
### Notes
- This release consolidates and improves batching behavior introduced in 4.4.20, with no breaking changes to the public API.
- All user-facing batching behavior is now governed by consistent memory and count constraints, improving predictability and stability during large dataset processing.
Replace calls to Rebatch(size) with RebatchBySize(obidefault.BatchMem(), obidefault.BatchSizeMax()) in batchiterator.go, fragment.go, and obirefidx.go to ensure consistent use of default memory and size limits for batch rebatching.
Introduce separate _BatchSize (min) and _BatchSizeMax (max) constants to replace the single _BatchSize variable. Update RebatchBySize to accept both maxBytes and maxCount parameters, flushing when either limit is exceeded. Set default batch size min to 1, max to 2000, and memory limit to 128 MB. Update CLI options and sequence_reader.go accordingly.
Eric Coissac