[Bug]: Under async scheduling, stateful logits processors that read output_token_ids values see -1 placeholders — logitsprocs_need_output_token_ids only counts CLI custom processors

[danbo610]

Your current environment

Observed on releases/v0.27.1 (6e448d0ea9), reading the same code paths on current main.
Runtime: vLLM 0.27.2.dev1 (source install), torch 2.13.0+cu130, 2x NVIDIA L20, TP=2, Python 3.12.
The issue is scheduling logic, not environment-specific.

🐛 Describe the bug

The v1 LogitsProcessor interface documents output_tok_ids as a live view of each request's generated tokens:

Key assumption: the output_tok_ids list (which is an element of each tuple in added) is a reference to the request's running output tokens list; via this reference, the logits processors always see the latest list of generated output tokens.

https://github.com/vllm-project/vllm/blob/6e448d0ea9/vllm/v1/sample/logits_processor/interface.py#L45-L48

Under async scheduling (default-on since 0.27), that assumption only holds conditionally. The worker-side lists are extended with -1 placeholders each step (real ids stay on the GPU in prev_sampled_token_ids):

https://github.com/vllm-project/vllm/blob/6e448d0ea9/vllm/v1/worker/gpu_model_runner.py#L3823-L3846

A repair mechanism exists precisely for this: set_async_sampled_token_ids / update_async_output_token_ids patch the -1s with real ids right before the logits processors run:

https://github.com/vllm-project/vllm/blob/6e448d0ea9/vllm/v1/worker/gpu_input_batch.py#L1030-L1056

But the repair is gated by needs_output_token_ids (penalties / bad_words / thinking budget / logitsprocs_need_output_token_ids):

https://github.com/vllm-project/vllm/blob/6e448d0ea9/vllm/v1/worker/gpu_input_batch.py#L908-L922

and the logitsprocs contribution to that gate only counts CLI-passed custom processors:

https://github.com/vllm-project/vllm/blob/6e448d0ea9/vllm/v1/worker/gpu_model_runner.py#L714-L726

# We currently don't know whether a particular custom logits processor
# uses output token ids so we set this conservatively. ...
logitsprocs_need_output_token_ids=bool(custom_logitsprocs),

So there are three under-/over-approximations of the same unknown:

1. Entry-point plugins (vllm.logits_processors group) are not counted — being fixed by [Bugfix] Include entry-point logits processor plugins in output token… #38199.
2. Builtin-registered processors are never counted. The three in-tree builtins happen to only read the list's length (MinTokensLogitsProcessor), never its values, so nothing in-tree notices. Any stateful processor registered via BUILTIN_LOGITS_PROCESSORS that reads values silently computes on placeholder garbage: full per-step overhead is paid, results are wrong, and there is no warning or error.
3. Conversely, CLI custom processors that don't read values still force the repair path (the conservative True), paying an unnecessary sync in async mode.

Reproduction / real-world impact

We integrated Google DeepMind's SynthID-Text watermarking (via the HF transformers processor) as a builtin stateful logits processor on top of releases/v0.27.1 — the first "reads values" consumer of this contract we know of:

• with --no-async-scheduling: 8/8 watermarked outputs detected, mean-g z-score avg 7.86 (range 5.51–11.47), 0 false positives on controls;
• with default async scheduling and identical code: 0/8 detected (z avg 0.39 ≈ control 0.33), while still paying ~11% per-request latency for the watermark compute — the context window degrades to [-1, -1, -1, -1].

Branch with the processor and a fail-safe guard (refuses to watermark and warns when async scheduling is detected): https://github.com/danbo610/vllm/tree/feature/synthid-watermark (see commit f18c17a3bb for the guard).

Minimal repro sketch: register any processor that prints output_tok_ids from apply() in BUILTIN_LOGITS_PROCESSORS, run with defaults (no penalties, no CLI --logits-processors), and observe [-1, -1, ...]; pass --no-async-scheduling (or any CLI custom processor, which flips the flag) and observe real ids.

Suggested fix

A declarative capability on the interface would solve all three cases at once, mirroring is_argmax_invariant():

class LogitsProcessor(ABC):
    @classmethod
    def needs_output_token_ids(cls) -> bool:
        return True  # conservative default for out-of-tree processors

• the three in-tree builtins override it to False (no behavior change, removes today's over-approximation for value-agnostic CLI procs);
• build_logitsprocs ORs the declarations across builtin + entry-point + CLI processors and feeds logitsprocs_need_output_token_ids;
• the interface.py "Key assumption" docstring gains a note that the live view is only guaranteed when this capability is declared (or async scheduling is off).

Short of that, amending the docstring plus a startup warning when async scheduling is enabled with non-in-tree processors would at least make the failure mode discoverable — today it is completely silent.

Happy to help validate a fix against the watermarking use case; we can also upstream the SynthID processor itself as a separate PR if there is interest.

[kakiuwang-ui]

I'd like to take this. Thanks for the very precise write-up — it made the three cases easy to pin down.

Plan

I'll implement the declarative capability you suggested, which fixes all three cases at once (including your builtin/SynthID case, which the entry-point-only fix does not cover):

1. interface.py — add a concrete classmethod on LogitsProcessor:

   @classmethod
   def needs_output_token_ids(cls) -> bool:
       return True  # conservative default for out-of-tree processors

   Plus a note on the BatchUpdate "Key assumption" docstring that the live view of values is only guaranteed when a processor declares this (or async scheduling is off).

2. builtin.py — the three in-tree builtins override to False. MinTokens only reads len(output_tok_ids) (correct even with -1 placeholders, since the list length still reflects the real token count); MinP/LogitBias don't read the list at all. No behavior change for them, and it removes today's over-approximation for value-agnostic CLI procs (case 3).

3. state.py — LogitsProcessors.__init__ ORs needs_output_token_ids() across every loaded processor (builtin + entry-point plugin + CLI), exposing a single needs_output_token_ids attribute.

4. gpu_model_runner.py — replace logitsprocs_need_output_token_ids=bool(custom_logitsprocs) with logitsprocs.needs_output_token_ids. Thinking-budget tracking stays dynamic (unchanged), so no interaction with reasoning_config there.

Net effect on the three cases:

• (1) entry-point plugins → counted, because they're instantiated in build_logitsprocs and default to True.
• (2) builtin-registered stateful processors (your SynthID case) → counted for the same reason; a value-agnostic builtin can opt out with return False.
• (3) value-agnostic CLI procs → can now declare False to skip the unnecessary async sync, instead of the blanket conservative True.

Relationship to #38199

#38199 addresses case (1) only, via a coarser LogitsProcessors.has_custom flag (true for any non-builtin processor). That leaves case (2) — the exact SynthID-as-builtin scenario in this issue — still broken, and doesn't help case (3). The per-processor declaration here is a superset, so I'll build on/supersede that narrower approach rather than duplicate it, and I'm happy to coordinate with the #38199 author on landing order.

Tests

Unit tests in tests/v1/logits_processors/test_correctness.py: (a) builtins declare False and the base default is True; (b) build_logitsprocs aggregates the gate correctly for builtins-only vs. a value-agnostic vs. a value-reading custom processor. I'll also run an async-scheduling e2e sanity check that a value-reading processor sees real ids rather than -1, and would gladly validate against the SynthID watermarking setup you linked.

I'll open the PR with duplicate-check notes and test commands/results.

Does this direction look right to you and the maintainers before I push?

[danbo610]

Direction looks right to me — glad to see it picked up, and I agree with your refinement: aggregating in LogitsProcessors.__init__ and exposing a single attribute is cleaner than what we did (we inlined the any() over logitsprocs.all at the InputBatch construction site in gpu_model_runner.py).

In fact we have already implemented and validated essentially this design on our branch — feel free to borrow anything useful (same license): danbo610@6341b66

Validation data from that branch (Qwen3.8-27B, TP=2 on NVIDIA L20, async scheduling on, SynthID declaring True):

• Watermark correctness restored: 8/8 watermarked outputs detected, mean-g z-score avg 8.77 (range 6.00–14.09); 0 false positives on unwatermarked controls (z avg 0.12). Identical code under async without the repair gate: 0/8, z avg 0.39.
• Repair-path cost is negligible for requests that don't opt in: non-watermarked 420-token requests run 16.2s — identical to pure async without repair (sync mode was 17.1s for the same requests). Only opted-in requests pay the watermark compute itself (~18.1s). This directly supports your case-3 point: value-agnostic processors declaring False is a real win, and value-reading ones declaring True doesn't tax the rest of the batch.

One small note from our implementation: we kept a defensive skip in the processor's apply() for any window still containing a negative id, as depth-of-defense for paths the repair doesn't cover. Probably worth a similar guard or assertion in your e2e test.

Happy to validate your PR against the SynthID setup once it's up — just ping me here. And if there's upstream interest in the watermarking processor itself, we'd propose it as a separate PR after this lands.