feat: limit concurrent open files during scatter #6

Merged
coissac merged 1 commits from push-rkytvkympxrn into main 2026-05-22 09:33:40 +00:00

1 Commits

Author SHA1 Message Date
Eric Coissac 85e1901898 feat: limit concurrent open files during scatter
Introduces a `max_open_files` CLI argument (default: 20) to cap concurrently open input files during scatter operations. The scatter phase now parallelizes sequence file partitioning across worker threads while enforcing a configurable concurrency limit using a custom semaphore and `GuardedIter` wrapper. This ensures bounded resource usage and prevents file handle exhaustion during index construction.
2026-05-22 11:28:44 +02:00