mirror of
https://github.com/metabarcoding/obitools4.git
synced 2026-04-30 12:00:39 +00:00
Eric Coissac
8c7017a99d
- Update obioptions.Version from "Release 4.4.29" to "/v/ Release v5" - Update version.txt from 4.29 → .30 (automated by Makefile)
1.5 KiB
1.5 KiB
Memory-Limited Biosequence Iterator
This Go function extends an IBioSequence iterator with memory-aware throttling to prevent excessive heap allocation during data processing.
Core Functionality
-
LimitMemory(fraction float64)
Returns a new iterator that respects an upper bound on heap usage relative to total system memory. -
Memory Monitoring
Usesruntime.ReadMemStats()andgithub.com/pbnjay/memory.TotalMemory()to compute the current heap fraction (Alloc / TotalMemory) dynamically. -
Backpressure Mechanism
While the memory fraction exceedsfraction, the producer goroutine yields control (runtime.Gosched()) until sufficient memory becomes available. -
Logging
Warns viaobilog.Warnfwhen:- Memory pressure persists (every ~1000 yields),
- Or wait duration becomes unusually long (>10,000 yielding cycles).
-
Concurrency Model
- A producer goroutine consumes from the original iterator and pushes items to
newIter, pausing as needed. - A dedicated consumer goroutine calls
WaitAndClose()to ensure graceful termination and resource cleanup.
- A producer goroutine consumes from the original iterator and pushes items to
Semantic Behavior
- Non-blocking consumer: Downstream consumers are not stalled; they read from an internal buffered channel (
newIter). - Adaptive rate control: The iterator automatically slows down when memory pressure rises, avoiding OOM conditions.
- Predictable resource use: Ensures heap usage stays below the specified
fraction(e.g., 0.5 → ≤ 50% of total RAM).