Fix slow batch generation in server by setting wired_limit#652
Merged
angeloskath merged 2 commits intoDec 5, 2025
Merged
Conversation
The server's batch generation path was missing the wired_limit setting that stream_generate uses, causing ~10x slower performance and ~30% GPU utilization on large models. When wired_limit is not set, model weights cannot stay pinned in GPU-accessible memory, causing frequent memory paging and GPU stalls. This fix sets max_recommended_working_set_size when creating the BatchGenerator and restores the previous limit when done, matching the behavior of stream_generate.
Contributor
Author
Merged
Contributor
Author
|
@angeloskath for visibility |
Member
|
We do need to set the wired limit but I don't want to do it this way in the server. It couples the generation and server in a way that would be good to avoid and adds complexity to the server logic which is already quite complex. I wonder if it could make sense to set wired limit when you construct the batch generator object and unset it when it goes out of scope? |
angeloskath
approved these changes
Dec 4, 2025
angeloskath
left a comment
angeloskath
left a comment
Member
There was a problem hiding this comment.
Looks great to me! Thanks for adding that :-)
Member
|
Would you mind updating |
- Set wired_limit in __init__() and restore in close() - Add __del__() as fallback safety net - Call close() explicitly in server and batch_generate - Remove redundant wired_limit context manager from batch_generate
otarkhan
force-pushed
the
fix-batch-generation-wired-limit
branch
from
9048adb to
ec8ba23
Compare
Contributor
Author
Good catch! Updated |
1 task
otarkhan
added a commit
to otarkhan/vllm-metal
that referenced
this pull request
Jan 8, 2026
Without wired_limit, model weights cannot stay pinned in GPU-accessible memory, causing frequent memory paging and GPU stalls. This results in ~30% GPU utilization instead of ~100%. The fix sets max_recommended_working_set_size as the wired limit before model loading, matching the fix in ml-explore/mlx-lm#652. Changes: - Set wired_limit in MetalWorker.init_device() for v1 engine - Set wired_limit in MetalModelRunner.load_model() for legacy path - Use new mx.set_wired_limit() API with fallback to deprecated API - Add error handling for graceful degradation
2 tasks
otarkhan
added a commit
to otarkhan/vllm-metal
that referenced
this pull request
Jan 8, 2026
Without wired_limit, model weights cannot stay pinned in GPU-accessible memory, causing frequent memory paging and GPU stalls. This results in ~30% GPU utilization instead of ~100%. The fix sets max_recommended_working_set_size as the wired limit before model loading, matching the fix in ml-explore/mlx-lm#652. Changes: - Set wired_limit in MetalWorker.init_device() for v1 engine - Set wired_limit in MetalModelRunner.load_model() for legacy path - Use new mx.set_wired_limit() API with fallback to deprecated API - Add error handling for graceful degradation Signed-off-by: otarkhan <osama.taha1994@gmail.com>
otarkhan
added a commit
to otarkhan/vllm-metal
that referenced
this pull request
Jan 8, 2026
Without wired_limit, model weights cannot stay pinned in GPU-accessible memory, causing frequent memory paging and GPU stalls. This results in ~30% GPU utilization instead of ~100%. The fix sets max_recommended_working_set_size as the wired limit before model loading, matching the fix in ml-explore/mlx-lm#652. Changes: - Set wired_limit in MetalWorker.init_device() for v1 engine - Set wired_limit in MetalModelRunner.load_model() for legacy path - Use new mx.set_wired_limit() API with fallback to deprecated API - Add error handling for graceful degradation Signed-off-by: otarkhan <osama.taha1994@gmail.com>
ericcurtin
pushed a commit
to vllm-project/vllm-metal
that referenced
this pull request
Jan 8, 2026
## Summary Without `wired_limit` set, model weights cannot stay pinned in GPU-accessible memory, causing frequent memory paging and GPU stalls. This results in ~30% GPU utilization instead of ~100%. This fix sets `max_recommended_working_set_size` as the wired limit before model loading, matching the fix in ml-explore/mlx-lm#652. ## Changes - Set `wired_limit` in `MetalWorker.init_device()` for v1 engine - Set `wired_limit` in `MetalModelRunner.load_model()` for legacy path - Use new `mx.set_wired_limit()` API with fallback to deprecated `mx.metal.set_wired_limit()` - Add error handling for graceful degradation ## Test plan - [x] Verified GPU utilization increases from ~30% to ~100% on Mac Studio with MiniMax-M2.1-Q8 - [x] Confirmed log message shows wired_limit being set on startup Signed-off-by: otarkhan <osama.taha1994@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
wired_limitsetting thatstream_generateuses, causing ~10x slower performance and ~30% GPU utilization on large models.wired_limitis not set, model weights cannot stay pinned in GPU-accessible memory, causing frequent memory paging and GPU stalls.max_recommended_working_set_sizewhen creating the BatchGenerator and restores the previous limit when done, matching the behavior ofstream_generate.Test plan
Before
Before.mp4
After
After.mp4