Compare commits

...

7 Commits

Author SHA1 Message Date
Eric Coissac
78e7933c6f 4.4.26: Static Linux Builds, Memory-Aware Batching & Cross-Compilation Fixes
This release includes critical build system improvements and enhanced batching capabilities for more predictable resource usage.

### Cross-Compilation & Static Builds
- Fixed cross-compilation for Linux by introducing architecture-specific `CGO_CFLAGS` (x86_64-linux-gnu and aarch64-linux-gnu), ensuring correct header resolution during static linking.
- Enabled fully static Linux binaries using musl, producing self-contained executables with no external runtime dependencies.

### Memory-Aware Batching (New Feature)
- Added `--batch-mem` CLI option to control batching based on estimated memory usage (e.g., 128K, 64M, 1G), in addition to size-based limits.
- Introduced configurable min/max batch sizes and memory thresholds, with conservative memory estimation per sequence to avoid over-allocation.
- Implemented intelligent flushing logic that triggers when *either* byte or record count limits are exceeded, ensuring predictable memory behavior.
- Improved garbage collection after large batch discards to reduce memory pressure during large-scale processing.

### Build & Toolchain Improvements
- Updated Go toolchain to 1.26.1 and bumped key dependencies (e.g., golang.org/x/net v0.38.0).
- Enhanced build error reporting: logs are now displayed before cleanup on failure.
- Fixed Makefile quoting for `LDFLAGS` containing spaces.
- Updated install script to properly configure GOROOT, GOPATH, and GOTOOLCHAIN, with added progress feedback for downloads.

All batching behavior is backward-compatible and uses sensible defaults (128 MB memory, min: 1 record, max: 2000 records) to ensure smooth upgrades.
2026-03-13 19:30:37 +01:00
Eric Coissac
afc9ffda85 chore: bump version to 4.4.25 and fix CGO_CFLAGS for cross-compilation
Update version to 4.4.25 in version.txt and pkg/obioptions/version.go.

Fix CGO_CFLAGS in release.yml by replacing generic '-I/usr/include' with architecture-specific paths (x86_64-linux-gnu and aarch64-linux-gnu) to ensure correct header inclusion during cross-compilation on Linux.
2026-03-13 19:30:29 +01:00
Eric Coissac
fdd972bbd2 fix: add CGO_CFLAGS for static Linux builds and update go.work.sum
- Add CGO_CFLAGS environment variable to release workflow for Linux builds
- Update go.work.sum with new golang.org/x/net v0.38.0 entry
- Remove obsolete logs archive file
2026-03-13 19:24:18 +01:00
coissac
76f595e1fe Merge pull request #95 from metabarcoding/push-kzmrqmplznrn
Version 4.4.24
2026-03-13 19:13:02 +01:00
coissac
1e1e5443e3 Merge branch 'master' into push-kzmrqmplznrn 2026-03-13 19:12:49 +01:00
coissac
58d685926b Merge pull request #94 from metabarcoding/push-lxxxlurqmqrt
4.4.23: Memory-aware batching, static Linux builds, and build improvements
2026-03-13 19:04:15 +01:00
Eric Coissac
e9f24426df 4.4.23: Memory-aware batching, static Linux builds, and build improvements
### Memory-Aware Batching
- Introduced configurable min/max batch size bounds and memory limits for precise resource control.
- Added `--batch-mem` CLI option to enable adaptive batching based on estimated sequence memory footprint (e.g., 128K, 64M, 1G).
- Implemented `RebatchBySize()` to handle both byte and count limits, flushing when either threshold is exceeded.
- Added conservative memory estimation via `BioSequence.MemorySize()` and enhanced garbage collection for explicit cleanup after large batch discards.
- Updated internal batching logic across core modules to consistently apply default memory (128 MB) and size (min: 1, max: 2000) bounds.

### Linux Build Enhancements
- Enabled static linking for Linux binaries using musl, producing portable, self-contained executables without external dependencies.

### Build System & Toolchain Improvements
- Updated Go toolchain to 1.26.1 with corresponding dependency bumps (e.g., go-getoptions, gval, regexp2, go-json, progressbar, logrus, testify).
- Fixed Makefile to safely quote LDFLAGS for paths with spaces.
- Improved build error handling: on failure, logs are displayed before cleanup and exit.
- Updated install script to correctly set GOROOT, GOPATH, and GOTOOLCHAIN, ensuring GOPATH directory creation.
- Added progress bar to curl downloads in the install script for visual feedback during Go and OBITools4 downloads.

All batching behavior remains non-breaking, with consistent constraints improving predictability during large dataset processing.
2026-03-13 19:03:50 +01:00
5 changed files with 6 additions and 2 deletions

View File

@@ -32,10 +32,12 @@ jobs:
goos: linux goos: linux
goarch: amd64 goarch: amd64
output_name: linux_amd64 output_name: linux_amd64
cgo_cflags: "-I/usr/include/x86_64-linux-gnu -I/usr/include"
- os: ubuntu-24.04-arm - os: ubuntu-24.04-arm
goos: linux goos: linux
goarch: arm64 goarch: arm64
output_name: linux_arm64 output_name: linux_arm64
cgo_cflags: "-I/usr/include/aarch64-linux-gnu -I/usr/include"
- os: macos-15-intel - os: macos-15-intel
goos: darwin goos: darwin
goarch: amd64 goarch: amd64
@@ -81,6 +83,7 @@ jobs:
GOARCH: ${{ matrix.goarch }} GOARCH: ${{ matrix.goarch }}
VERSION: ${{ steps.get_version.outputs.version }} VERSION: ${{ steps.get_version.outputs.version }}
CC: ${{ matrix.goos == 'linux' && 'musl-gcc' || '' }} CC: ${{ matrix.goos == 'linux' && 'musl-gcc' || '' }}
CGO_CFLAGS: ${{ matrix.cgo_cflags || '' }}
run: | run: |
if [ "$GOOS" = "linux" ]; then if [ "$GOOS" = "linux" ]; then
make LDFLAGS='-linkmode=external -extldflags=-static' obitools make LDFLAGS='-linkmode=external -extldflags=-static' obitools

View File

@@ -52,6 +52,7 @@ golang.org/x/image v0.6.0/go.mod h1:MXLdDR43H7cDJq5GEGXEVeeNhPgi+YYEQ2pC1byI1x0=
golang.org/x/mod v0.13.0 h1:I/DsJXRlw/8l/0c24sM9yb0T4z9liZTduXvdAWYiysY= golang.org/x/mod v0.13.0 h1:I/DsJXRlw/8l/0c24sM9yb0T4z9liZTduXvdAWYiysY=
golang.org/x/mod v0.13.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c= golang.org/x/mod v0.13.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/mod v0.14.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c= golang.org/x/mod v0.14.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/net v0.38.0/go.mod h1:ivrbrMbzFq5J41QOQh0siUuly180yBYtLp+CKbEaFx8=
golang.org/x/net v0.52.0/go.mod h1:R1MAz7uMZxVMualyPXb+VaqGSa3LIaUqk0eEt3w36Sw= golang.org/x/net v0.52.0/go.mod h1:R1MAz7uMZxVMualyPXb+VaqGSa3LIaUqk0eEt3w36Sw=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4 h1:uVc8UZUe6tr40fFVnUP5Oj+veunVezqYl9z7DYw9xzw= golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4 h1:uVc8UZUe6tr40fFVnUP5Oj+veunVezqYl9z7DYw9xzw=
golang.org/x/text v0.13.0 h1:ablQoSUd0tRdKxZewP80B+BaqeKJuVhuRxj/dkrun3k= golang.org/x/text v0.13.0 h1:ablQoSUd0tRdKxZewP80B+BaqeKJuVhuRxj/dkrun3k=

Binary file not shown.

View File

@@ -3,7 +3,7 @@ package obioptions
// Version is automatically updated by the Makefile from version.txt // Version is automatically updated by the Makefile from version.txt
// The patch number (third digit) is incremented on each push to the repository // The patch number (third digit) is incremented on each push to the repository
var _Version = "Release 4.4.24" var _Version = "Release 4.4.26"
// Version returns the version of the obitools package. // Version returns the version of the obitools package.
// //

View File

@@ -1 +1 @@
4.4.24 4.4.26