[Bug]: A multimodal item can still be split across prefill chunks with disable_chunked_mm_input=True

[chrisc36]

Your current environment

The output of python collect_env.py

GPU0 X SYS 3-63,131-191 0 N/A NIC0 SYS X

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_bond_0

==============================
Environment Variables

NVIDIA_VISIBLE_DEVICES=GPU-216bfdad-f232-5610-63d8-c1754594308c
VLLM_BUILD_URL=https://buildkite.com/vllm/release-v2/builds/4244
NVIDIA_REQUIRE_CUDA=cuda>=12.9 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=560,driver<561 brand=grid,driver>=560,driver<561 brand=tesla,driver>=560,driver<561 brand=nvidia,driver>=560,driver<561 brand=quadro,driver>=560,driver<561 brand=quadrortx,driver>=560,driver<561 brand=nvidiartx,driver>=560,driver<561 brand=vapps,driver>=560,driver<561 brand=vpc,driver>=560,driver<561 brand=vcs,driver>=560,driver<561 brand=vws,driver>=560,driver<561 brand=cloudgaming,driver>=560,driver<561 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566 brand=unknown,driver>=570,driver<571 brand=grid,driver>=570,driver<571 brand=tesla,driver>=570,driver<571 brand=nvidia,driver>=570,driver<571 brand=quadro,driver>=570,driver<571 brand=quadrortx,driver>=570,driver<571 brand=nvidiartx,driver>=570,driver<571 brand=vapps,driver>=570,driver<571 brand=vpc,driver>=570,driver<571 brand=vcs,driver>=570,driver<571 brand=vws,driver>=570,driver<571 brand=cloudgaming,driver>=570,driver<571
TORCH_CUDA_ARCH_LIST=7.5 8.0 8.6 8.9 9.0 10.0 12.0
NVIDIA_DRIVER_CAPABILITIES=compute,utility
VLLM_IMAGE_TAG=vllm/vllm-openai:v0.26.0-cu129-ubuntu2404
VLLM_USAGE_SOURCE=production-docker-image
CUDA_VERSION=12.9.1
VLLM_ENABLE_CUDA_COMPATIBILITY=0
VLLM_BUILD_PIPELINE=019d130e-464e-4ff7-b84b-492992c0c06b
LD_LIBRARY_PATH=/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/cuda/lib64
OMP_NUM_THREADS=8
VLLM_BUILD_COMMIT=ffd46bfab2128bb84146050e98b51a617c6575ab
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root

🐛 Describe the bug

I think there is an off-by-one error in the scheduler's self.scheduler_config.disable_chunked_mm_input logic.

_try_schedule_encoder_inputs in Scheduler guards against splitting a multimodal item across prefill chunks like this:

if (
    self.scheduler_config.disable_chunked_mm_input
    and num_computed_tokens < start_pos                                            # (A)
    and (num_computed_tokens + num_new_tokens) < (start_pos + num_encoder_tokens)  # (B)
):
    num_new_tokens = max(0, start_pos - (num_computed_tokens + shift_computed_tokens))
    break

Check (A) is causing this statement to incorrectly return False when num_computed_tokens == start_pos.
If num_computed_tokens == start_pos then the tokens after num_computed_tokens are exactly
the start of the next multi-modal item, but (A) returns False so the (B) isn't checked and the
multi-modal item might get chunked.

This issue is made worse by the fact if the check does fire when num_computed_tokens < start_pos then
num_computed_tokens will become start_pos, and we will therefore hit this issue if we can't
fit the multi-modal item into the token budget on the second attempt.

This can silently break models with bidirectional attention, since incorrect multi-modal
splitting will prevent tokens that should be able to cross-attend from cross-attending.

Reproduction

We can observe the error by testing the function directly:

from types import SimpleNamespace
from vllm.v1.core.sched.scheduler import Scheduler

START, LENGTH = 3, 766

def request():
    # Fake multimodal item of length `LENGTH` offset at `START`
    pos = SimpleNamespace(offset=START, length=LENGTH, is_embed=None,
                          get_num_embeds=lambda: LENGTH,
                          get_embeds_indices_in_range=lambda a, b: (a, b))
    return SimpleNamespace(has_encoder_inputs=True, request_id="r0",
        mm_features=[SimpleNamespace(mm_position=pos, identifier="img0", modality="image")])

def sched():
    # Fake scheduler `self` with disable_chunked_mm_input=True
    return SimpleNamespace(
        scheduler_config=SimpleNamespace(disable_chunked_mm_input=True),
        is_encoder_decoder=False, ec_connector=None,
        encoder_cache_manager=SimpleNamespace(
            check_and_update_cache=lambda r, i: False,
            can_allocate=lambda r, i, b, n: True))

def ask(num_computed_tokens, num_new_tokens):
    # Ask how many tokens the scheduler will actually let this request run this step.
    # given we have done `num_computed_tokens` and can afford to do up to `num_new_tokens` more
    return Scheduler._try_schedule_encoder_inputs(
        sched(), request(), num_computed_tokens, num_new_tokens,
        100_000,   # stubbed encoder_compute_budget
        0,         # stubbed shift_computed_tokens
    )[1]

# Works: We can don't have room for the image so it only allows 3 new tokens
print(ask(0, 500))                      # 3

# Breaks: the scheduler says we can go ahead and compute up to token 500, but this will
# split the image since it ends on position 766
print(ask(3, 500))                  # 500

Fix

As far as I can tell changing it to num_computed_tokens <= start_pos will fix the issue

Claude thinks (A) can be deleted. I am hesitant to endorse that since I don't fully understand the
logic and I am not sure why (A) was put there in the first place, but I here is its reasoning:

(A) can also just be deleted, and I believe that is equivalent in normal operation. If the guard is
working then the only reachable states are:

num_computed_tokens	meaning	(B)
< start_pos	before the item	decides
== start_pos	at the item, nothing of it computed	decides
>= start_pos + length	item already fully computed	already False, since the left side alone is >= the right

So (B) rules out the "already processed" case by itself and (A) adds nothing except excluding the
boundary. The only state where the two forms differ is start_pos < num_computed_tokens < start_pos + length, i.e. resuming inside an item, which I think is only reachable with prefix
caching. Deferring there seems right too for a bidirectional model -- the remainder of the item
should not be split either -- but that path is reasoned about rather than tested, so <= is the
change I would suggest.

Either way it should not be possible to starve a request: compute_mm_encoder_budget guarantees
max_num_batched_tokens >= max_tokens_per_mm_item whenever disable_chunked_mm_input is set, so a
request scheduled first in a step always has room for a whole item, and a request given zero tokens
hits continue rather than break, so the rest of the batch keeps moving.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[adityaanikam]

Confirmed the exact arithmetic: with num_computed_tokens=3 and start_pos=3, check (A) evaluates 3 < 3, False, so the whole guard short circuits before (B) ever gets evaluated, even though this is exactly the state where nothing of the item has been computed yet and (B) would have caught the overlap. Verified against the reachable state table you laid out too, once num_computed_tokens crosses into start_pos < num_computed_tokens < start_pos + length, that's resuming inside an item, which only prefix caching can reach, and I don't have a way to exercise that path here either.

Your <= suggestion is the safer of the two fixes since it only changes the boundary rather than removing a check outright. Would you rather see that go in as is, or is there someone who actually knows why (A) was added originally who should weigh in before it's simplified further?

[chrisc36]

I don't really have a preference, as you say <= seems safest.

I will differ to people with more experience with vLLM on whether the (A) check can be cleaned completely or if its worth the effort of to try and do so.

[DavinZhang]

Thanks for the unusually precise report — the reachable-state table made this quick to verify. I reproduced it and then pushed the same scenario through the real scheduler; posting what I found plus a preference on (A), in case it is useful to whoever picks this up.

Setup: commit 83d4c6196ad90f98515c6be113cb2e957b60a832 (the guard is still identical on main today). Everything below is scheduler-level, no GPU compute needed, so it is cheap to re-run.

1. The boundary reproduces exactly as described

Your stub gives ask(3, 500) -> 500 verbatim. I also wanted to know which real configurations can park num_computed_tokens on start_pos, since compute_mm_encoder_budget() already rules out the obvious "budget smaller than one item" case. Two ways in:

• another request in the same step eating the token budget (transient), and
• --long-prefill-token-threshold capping per-step tokens below the item length (structural — that cap is applied to num_new_tokens before this guard runs).

llava-1.5-7b, max_num_batched_tokens=800, long_prefill_token_threshold=770, one text-only request of 3000 tokens plus one request with a single mm item at [3, 769):

step 1: scheduled={'text': 770, 'mm': 3}    mm_req: computed 0 -> 3     # (A) fires, correct rollback
step 2: scheduled={'text': 770, 'mm': 30}   mm_req: computed 3 -> 33    # <-- item chunked
step 3: scheduled={'text': 770, 'mm': 30}   mm_req: computed 33 -> 63   # <-- item chunked
step 4: scheduled={'text': 690, 'mm': 110}  mm_req: computed 63 -> 173  # <-- item chunked

That is exactly the "made worse" mechanism from your report: the rollback parks num_computed_tokens on start_pos, so the very next step is guaranteed to hit the off-by-one. With <=, the same scenario schedules 0 tokens for that request in steps 2-4 and then the whole item in step 5 (computed 3 -> 773): no chunking, no starvation.

2. Preference: <=, and I would keep (A)

I agree with @adityaanikam that <= is the safer of the two, and I have a concrete reason to keep (A) rather than delete it: its strict < is load-bearing for encoder-decoder models, by accident. Changing only < to <= makes three cross-attention tests in tests/v1/core/test_scheduler.py fail with an empty scheduled_new_reqs (assert 0 == 3):

• test_cross_attn_blocks_not_over_allocated
• test_cross_attn_blocks_not_under_allocated
• test_variable_length_cross_attn_block_allocation

For encoder-decoder models start_pos is only a marker (always 0) and mm_position.length is the encoder length, which is not part of the decoder token sequence — so the range arithmetic in this guard has no meaning there. x < 0 was keeping those models out of the guard silently; x <= 0 lets them in and rolls them back to zero tokens on every step. (Your stub sets is_encoder_decoder=False, so it cannot surface this.) So I think the guard wants an explicit and not self.is_encoder_decoder next to the <=.

3. A correction to the "cannot starve" argument

compute_mm_encoder_budget() validates max_num_batched_tokens, but nothing validates long_prefill_token_threshold. With long_prefill_token_threshold=500 and max_tokens_per_mm_item=576 (llava-1.5-7b) the per-step budget never grows, so:

• on main today the item is silently chunked (split at steps 2-3 in my trace), and
• with the <= fix alone the request gets 0 tokens every step, forever:

step 1: scheduled={'text': 500, 'mm': 3}  mm_req: computed 0 -> 3
step 6: scheduled={'text': 500}           mm_req: computed 3 -> 3
step 7: scheduled={}                      mm_req: computed 3 -> 3   # engine idle from here on
step 8: scheduled={}                      mm_req: computed 3 -> 3

Also, the zero-token paths differ between the two call sites: the running-queue one does continue (scheduler.py:567), but the waiting-queue one does break (scheduler.py:951), so a waiting request stuck at zero additionally head-of-line-blocks the requests behind it.

Two ways to handle that configuration: keep chunking it (silently violating the flag), or reject it at startup. I lean towards rejecting, right next to the existing max_num_batched_tokens check, since the entire point of the flag is that the item is not split. The cost is real and worth stating: a configuration that combines a small long_prefill_token_threshold with disable_chunked_mm_input would go from "silently wrong" to "does not start".

4. Regression check

tests/v1/core/test_scheduler.py: 147 passed before, 149 after (the two added tests). Whole tests/v1/core directory: 465 -> 467 passed with a byte-identical set of pre-existing failures on both trees (32 failed / 2 errors in my sandbox, all model-download or engine-spawning tests, unrelated to this change). Old code with the new tests fails exactly the two new ones (mm item [3, 769) was chunked: 3 -> 33, and DID NOT RAISE ValueError).

Patch I have been testing — guard fix, config validation, two regression tests

--- a/vllm/v1/core/sched/scheduler.py
+++ b/vllm/v1/core/sched/scheduler.py
@@ -1573,7 +1573,11 @@
             # only cover part of the mm input, roll back to before the mm item.
             if (
                 self.scheduler_config.disable_chunked_mm_input
-                and num_computed_tokens < start_pos
+                # For encoder-decoder models start_pos is only a marker (always 0)
+                # and the encoder tokens are not part of the decoder sequence, so
+                # the range arithmetic below does not apply.
+                and not self.is_encoder_decoder
+                and num_computed_tokens <= start_pos
                 and (num_computed_tokens + num_new_tokens)
                 < (start_pos + num_encoder_tokens)
             ):
--- a/vllm/v1/core/encoder_cache_manager.py
+++ b/vllm/v1/core/encoder_cache_manager.py
@@ -306,16 +306,26 @@
 
     max_tokens_per_mm_item = max(mm_max_toks_per_item.values())
 
-    if (
-        scheduler_config.disable_chunked_mm_input
-        and max_tokens_per_mm_item > scheduler_config.max_num_batched_tokens
-    ):
-        raise ValueError(
-            "Chunked MM input disabled but max_tokens_per_mm_item "
-            f"({max_tokens_per_mm_item}) is larger than max_num_batched_tokens"
-            f" ({scheduler_config.max_num_batched_tokens}). Please increase "
-            "max_num_batched_tokens."
-        )
+    if scheduler_config.disable_chunked_mm_input:
+        # A multimodal item has to fit into a single batch, so every per-step cap
+        # on the number of scheduled tokens must be able to hold the largest item.
+        # Otherwise the scheduler can never schedule the item as a whole and the
+        # request would be stuck forever.
+        if max_tokens_per_mm_item > scheduler_config.max_num_batched_tokens:
+            raise ValueError(
+                "Chunked MM input disabled but max_tokens_per_mm_item "
+                f"({max_tokens_per_mm_item}) is larger than max_num_batched_tokens"
+                f" ({scheduler_config.max_num_batched_tokens}). Please increase "
+                "max_num_batched_tokens."
+            )
+        if 0 < scheduler_config.long_prefill_token_threshold < max_tokens_per_mm_item:
+            raise ValueError(
+                "Chunked MM input disabled but max_tokens_per_mm_item "
+                f"({max_tokens_per_mm_item}) is larger than "
+                "long_prefill_token_threshold "
+                f"({scheduler_config.long_prefill_token_threshold}). Please "
+                "increase long_prefill_token_threshold."
+            )
 
     encoder_compute_budget = max(
         scheduler_config.max_num_encoder_input_tokens, max_tokens_per_mm_item
--- a/tests/v1/core/test_scheduler.py
+++ b/tests/v1/core/test_scheduler.py
@@ -591,6 +591,88 @@
         )
 
 
+def test_no_mm_input_chunking_when_stopped_at_mm_item_start():
+    """Regression test for the boundary where the previous step stopped exactly
+    at the first token of the multimodal item.
+
+    A long text request eats most of the per-step token budget, so the mm
+    request is first rolled back to the token right before the item
+    (num_computed_tokens == mm offset). From that position the remaining budget
+    is still too small for the whole item, and the item must not be chunked.
+    """
+    item_start, item_length = 3, 766
+    item_end = item_start + item_length
+    scheduler = create_scheduler(
+        model="llava-hf/llava-1.5-7b-hf",
+        max_num_batched_tokens=800,
+        # Must be >= item_length, otherwise the item never fits in a step and
+        # the config is rejected (see the test below).
+        long_prefill_token_threshold=770,
+        disable_chunked_mm_input=True,
+        max_model_len=4096,
+    )
+    requests = create_requests(
+        num_requests=2,
+        num_tokens=3000,
+        mm_positions=[[], [PlaceholderRange(offset=item_start, length=item_length)]],
+        max_tokens=64,
+    )
+    for request in requests:
+        scheduler.add_request(request)
+    mm_request = requests[1]
+
+    # Step 1 rolls back to the item start; steps 2-4 cannot fit the item and
+    # must schedule nothing for the mm request; by step 5 the text request is
+    # done prefilling and the whole item is scheduled at once.
+    scheduled_per_step = []
+    for _ in range(5):
+        num_computed_tokens = mm_request.num_computed_tokens
+        output = scheduler.schedule()
+        num_scheduled_tokens = output.num_scheduled_tokens.get(
+            mm_request.request_id, 0
+        )
+        scheduled_per_step.append(num_scheduled_tokens)
+        # The scheduled range must never end strictly inside the mm item.
+        assert not (
+            item_start < num_computed_tokens + num_scheduled_tokens < item_end
+        ), (
+            f"mm item [{item_start}, {item_end}) was chunked: "
+            f"{num_computed_tokens} -> "
+            f"{num_computed_tokens + num_scheduled_tokens}"
+        )
+        if not output.num_scheduled_tokens:
+            continue
+        req_ids = list(output.num_scheduled_tokens)
+        scheduler.update_from_output(
+            output,
+            ModelRunnerOutput(
+                req_ids=req_ids,
+                req_id_to_index={r: i for i, r in enumerate(req_ids)},
+                sampled_token_ids=[[] for _ in req_ids],
+                logprobs=None,
+                prompt_logprobs_dict={},
+                pooler_output=[],
+            ),
+        )
+
+    assert scheduled_per_step == [item_start, 0, 0, 0, 770]
+    # And the item is not starved: it is fully computed once the budget frees up.
+    assert mm_request.num_computed_tokens >= item_end
+
+
+def test_no_mm_input_chunking_rejects_small_long_prefill_threshold():
+    # long_prefill_token_threshold caps the tokens scheduled per step, so a
+    # threshold below the size of one mm item makes the item unschedulable.
+    with pytest.raises(ValueError):
+        _ = create_scheduler(
+            model="llava-hf/llava-1.5-7b-hf",
+            max_num_batched_tokens=1024,
+            long_prefill_token_threshold=500,
+            disable_chunked_mm_input=True,
+            max_model_len=2048,
+        )
+
+
 @pytest.mark.parametrize("enable_prefix_caching", [True, False])
 def test_schedule_concurrent_partial_requests(enable_prefix_caching: bool):
     """Test scheduling behavior with concurrent partial requests.

I am happy to open this as a PR if the direction looks right to the maintainers — the startup ValueError in §3 is the only user-visible behaviour change, so that is the part worth a second opinion. If you would rather keep the fix to the <= plus the encoder-decoder guard and handle the threshold validation separately, that works as well; the two tests split cleanly along the same line.