Your current environment
NA
🐛 Describe the bug
Problem
In examples/disaggregated/disaggregated_encoder/disagg_epd_proxy.py, encoder requests are distributed with:
# Round-robin over encode servers to distribute load a bit
url_cycle = (e_urls[i % len(e_urls)] for i in range(len(mm_items)))
This is only round-robin within a single incoming request. Since i always starts at 0 for every request, e_urls[0] receives the first multimodal item of every request.
If most requests contain one image/audio item, then all encoder traffic goes to the first encoder instance. If most requests contain fewer multimodal items than encoder instances, later encoder instances receive little or no traffic.
Example
With 3 encoder instances:
Request with 1 image -> E0
Request with 1 image -> E0
Request with 1 image -> E0
With 2 images per request:
Request 1 -> E0, E1
Request 2 -> E0, E1
Request 3 -> E0, E1
E2 is never selected in the second case.
Expected Behavior
The proxy should distribute encoder sub-requests fairly across encoder instances over time, not restart from e_urls[0] for every incoming request.
Possible Fixes
One option is to maintain a global round-robin cursor, for example:
start = app.state.encoder_rr_idx
app.state.encoder_rr_idx = (app.state.encoder_rr_idx + len(mm_items)) % len(e_urls)
url_cycle = (e_urls[(start + i) % len(e_urls)] for i in range(len(mm_items)))
Because the proxy is async, this should probably be protected with an asyncio.Lock or replaced with another concurrency-safe strategy.
Another simpler option is to choose an encoder randomly per multimodal item:
target_url = random.choice(e_urls)
This avoids hot-spotting e_urls[0], although it is not deterministic round-robin.
Affected File
examples/disaggregated/disaggregated_encoder/disagg_epd_proxy.py
Additional Context
The current comment says “Round-robin over encode servers,” but the implementation is only per-request modulo assignment. This can lead to uneven encoder load in common single-image request workloads.
Before submitting a new issue...
Your current environment
NA
🐛 Describe the bug
Problem
In
examples/disaggregated/disaggregated_encoder/disagg_epd_proxy.py, encoder requests are distributed with:This is only round-robin within a single incoming request. Since
ialways starts at0for every request,e_urls[0]receives the first multimodal item of every request.If most requests contain one image/audio item, then all encoder traffic goes to the first encoder instance. If most requests contain fewer multimodal items than encoder instances, later encoder instances receive little or no traffic.
Example
With 3 encoder instances:
With 2 images per request:
E2is never selected in the second case.Expected Behavior
The proxy should distribute encoder sub-requests fairly across encoder instances over time, not restart from
e_urls[0]for every incoming request.Possible Fixes
One option is to maintain a global round-robin cursor, for example:
Because the proxy is async, this should probably be protected with an
asyncio.Lockor replaced with another concurrency-safe strategy.Another simpler option is to choose an encoder randomly per multimodal item:
This avoids hot-spotting
e_urls[0], although it is not deterministic round-robin.Affected File
examples/disaggregated/disaggregated_encoder/disagg_epd_proxy.pyAdditional Context
The current comment says “Round-robin over encode servers,” but the implementation is only per-request modulo assignment. This can lead to uneven encoder load in common single-image request workloads.
Before submitting a new issue...