fix(engine): stop double-subtracting pool size in calcAgents#674
Open
LaGrunge wants to merge 1 commit into
Open
fix(engine): stop double-subtracting pool size in calcAgents#674LaGrunge wants to merge 1 commit into
LaGrunge wants to merge 1 commit into
Conversation
availableAgents already represents the pool's current total capacity (free + running), so adding availablePoolAgents again on top of it in reqPoolAgents made the target permanently short by the pool's own size. Under load (no idle agents, non-empty pending queue) this drove reqPoolAgents negative, so the create-agents branch never ran and the autoscaler stalled at its current size instead of scaling up.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is it about?
calcAgents()computed the required agent-count delta as:availableAgents(freeTasks + runningTasks) already represents thepool's current total capacity, so adding
availablePoolAgentsagaindouble-counted the existing pool size. Under sustained load — zero
idle agents and a non-empty pending queue — this drove
reqPoolAgentsnegative, so the create-agents branch never ran: the autoscaler
appeared to stall at its current size instead of scaling up, even with
plenty of headroom under
WOODPECKER_MAX_AGENTS.The fix removes the redundant term:
Why this was hard to notice
The bug is silent whenever there's any idle capacity or the queue is
empty — the
maxDown/maxUpclamps happen to produce the same resultin those cases either way. It only manifests when the pool is fully
busy and a backlog builds up, which is exactly the situation where
autoscaling matters most.
Testing
should create new agents when pool is fully busy with a backlog) reproducing the failure: 3 busy agents +2 pending workflows previously computed
-1(wanting to shrink);now correctly computes
+2.