mirror of
https://github.com/metabarcoding/obitools4.git
synced 2026-04-30 03:50:39 +00:00
⬆️ version bump to v4.5
- Update obioptions.Version from "Release 4.4.29" to "/v/ Release v5" - Update version.txt from 4.29 → .30 (automated by Makefile)
This commit is contained in:
@@ -0,0 +1,29 @@
|
||||
# Memory-Limited Biosequence Iterator
|
||||
|
||||
This Go function extends an `IBioSequence` iterator with memory-aware throttling to prevent excessive heap allocation during data processing.
|
||||
|
||||
## Core Functionality
|
||||
|
||||
- **`LimitMemory(fraction float64)`**
|
||||
Returns a new iterator that respects an upper bound on heap usage relative to total system memory.
|
||||
|
||||
- **Memory Monitoring**
|
||||
Uses `runtime.ReadMemStats()` and `github.com/pbnjay/memory.TotalMemory()` to compute the current heap fraction (`Alloc / TotalMemory`) dynamically.
|
||||
|
||||
- **Backpressure Mechanism**
|
||||
While the memory fraction exceeds `fraction`, the producer goroutine yields control (`runtime.Gosched()`) until sufficient memory becomes available.
|
||||
|
||||
- **Logging**
|
||||
Warns via `obilog.Warnf` when:
|
||||
- Memory pressure persists (every ~1000 yields),
|
||||
- Or wait duration becomes unusually long (>10,000 yielding cycles).
|
||||
|
||||
- **Concurrency Model**
|
||||
- A producer goroutine consumes from the original iterator and pushes items to `newIter`, pausing as needed.
|
||||
- A dedicated consumer goroutine calls `WaitAndClose()` to ensure graceful termination and resource cleanup.
|
||||
|
||||
## Semantic Behavior
|
||||
|
||||
- **Non-blocking consumer**: Downstream consumers are not stalled; they read from an internal buffered channel (`newIter`).
|
||||
- **Adaptive rate control**: The iterator automatically slows down when memory pressure rises, avoiding OOM conditions.
|
||||
- **Predictable resource use**: Ensures heap usage stays below the specified `fraction` (e.g., 0.5 → ≤ 50% of total RAM).
|
||||
Reference in New Issue
Block a user