Introduce concurrency-bounded parallel processing for index cache layers

Implement an `obipipeline::Throttle` with an RAII guard to acquire and release concurrency slots. Expose new bounded parallel methods on `IndexCache` to process cached layers with a configurable cap. Refactor downstream aggregation logic to use single-pass parallel map-reduce instead of manual collect-map-reduce sequences, enforcing a maximum of 8 concurrent layer scans to bound memory usage.

This commit is contained in:

Eric Coissac

2026-08-28 20:20:12 +02:00

parent 9dee6dcd08

commit d084396aba

6 changed files with 211 additions and 62 deletions

src/Cargo.lock

Generated

View File

@@ -1567,6 +1567,7 @@ dependencies = [
  "obicompactvec",
  "obikindex",
  "obikseq",
  "obipipeline",
  "rayon",
  "tempfile",
 ]