Skip to content

Commit 020f291

Browse files
authored
fix: prevent batcher activity deadlock when retrying failed tasks (#10630) (#10657)
## What changed? Backports #10630 to `cloud/v1.32.0-157`. Workers now retry retryable task failures in place and emit exactly one response per task instead of re-queuing onto the worker-only task channel; the heartbeat resume point tracks the oldest incomplete page; the worker count is clamped to at least 1; and the loop stops submitting empty tasks once a page is fully submitted. ## Why? Patch the 1.32.0-157 release: re-queuing retries onto the worker-only `taskCh` could deadlock the worker pool under a burst of retryable errors, leaving the admin/batch activity heartbeating with no progress. Adapted, not a clean cherry-pick: 1.32.0-157 predates #10627, so this targets the pre-#10627 batcher structure (and converts the touched `errors.New(fmt.Sprintf)` sites to `fmt.Errorf` plus `eventId`→`eventID` to satisfy the linters on the moved code). ## How did you test it? - [x] built (`go build ./service/worker/batcher/...`) - [x] covered by existing tests - [x] added new unit test(s) ## Potential risks Retries hold a worker slot while retrying, bounded by AttemptsOnRetryableError. Scope limited to admin/batch operation processing.
1 parent 18f4567 commit 020f291

2 files changed

Lines changed: 358 additions & 219 deletions

File tree

0 commit comments

Comments
 (0)