fix(models): raise GenerationError for streamed refusals - #1992
Open
nzinci wants to merge 1 commit into
Open
Conversation
Check refusal deltas before yielding content from OpenAI-compatible streaming APIs. Keep first-choice behavior and support SDK versions that do not define the refusal field. Add sync and async coverage for OpenAI, vLLM, and SGLang.
|
📚 Documentation preview: https://dottxt-ai.github.io/outlines/pr-preview/pr-1992/ Preview updates automatically with each commit. |
Contributor
Author
|
Tests and coverage pass. The style check only fails because mypy cannot find |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The sync and async stream paths for OpenAI, vLLM, and SGLang only looked at
delta.content. If a provider returned a refusal together with content, Outlines yielded the content instead of raising theGenerationErrorused by non-streamed calls.This checks the refusal before yielding content in all six paths. It uses
getattrbecause older OpenAI SDK versions do not define the refusal field, and it leaves the existing first-choice behavior unchanged.The tests cover refusal chunks that also contain content, empty choices, missing refusal attributes, and refusals in later choices.
Tests
uv run pytest -q tests/models/test_provider_exceptions.py -k refusal— 18 passed, 62 deselecteduv run pytest -q tests/models/test_openai.py tests/models/test_vllm.py tests/models/test_sglang.py -m 'not api_call'— 54 passed, 19 deselecteduv run pytest -q tests/models/test_provider_exceptions.py— 80 passedgit diff --check— passed