refactor: optimize dump partition iteration and add progress tracking #20

2026-06-09T09:34:04Z

coissac commented

2026-06-09 09:34:04 +00:00

Refactor partition iteration to support a generic on_partition callback executed after each parallel partition completes. Split the logic into bounded and unbounded paths; the bounded path uses an AtomicUsize to enforce row limits, while the unbounded path eliminates atomic contention to improve throughput. Additionally, integrate a progress bar into the dump command by passing an increment callback to idx.dump(), ensuring proper initialization and cleanup.

Refactor partition iteration to support a generic `on_partition` callback executed after each parallel partition completes. Split the logic into bounded and unbounded paths; the bounded path uses an `AtomicUsize` to enforce row limits, while the unbounded path eliminates atomic contention to improve throughput. Additionally, integrate a progress bar into the dump command by passing an increment callback to `idx.dump()`, ensuring proper initialization and cleanup.

coissac added 1 commit 2026-06-09 09:34:05 +00:00

refactor: optimize dump partition iteration and add progress tracking db730e9cf6

Refactor partition iteration to support a generic `on_partition` callback executed after each parallel partition completes. Split the logic into bounded and unbounded paths; the bounded path uses an `AtomicUsize` to enforce row limits, while the unbounded path eliminates atomic contention to improve throughput. Additionally, integrate a progress bar into the dump command by passing an increment callback to `idx.dump()`, ensuring proper initialization and cleanup.

coissac merged commit b0dab452f6 into main

2026-06-09 09:34:13 +00:00

coissac deleted branch push-xqswlxlvmyrq

2026-06-09 09:34:14 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: OBIKmers/obikmer#20