Skip to content

Fix post-compact 400 on SGLang/vLLM: merge leading system messages - #860

Merged
mpfaffenberger merged 1 commit into
mpfaffenberger:mainfrom
thomwebb:fix/merge-leading-system-messages
Aug 24, 2026
Merged

Fix post-compact 400 on SGLang/vLLM: merge leading system messages#860
mpfaffenberger merged 1 commit into
mpfaffenberger:mainfrom
thomwebb:fix/merge-leading-system-messages

Conversation

@thomwebb

Copy link
Copy Markdown
Collaborator

Summary

Resuming a session after auto-compact on SGLang / vLLM-style OpenAI-compatible endpoints fails with:

400: {'message': 'System message must be at the beginning.', 'type': 'BadRequest'}

Root cause

After SummarizingCompaction runs, the compacted history opens with a summary SystemPromptPart (produced by pydantic-ai-harness). On the next request, pydantic-ai's OpenAIChatModel._map_messages also inserts the agent's per-turn instruction_parts right after leading system messages — so the wire format carries two leading system messages. Strict backends accept only one.

pydantic-ai already ships _merge_leading_system_messages for exactly this case, but it only fires when the model profile sets openai_chat_supports_multiple_system_messages=False — and none of our OpenAI-compatible model types did.

Fix

New _strict_openai_profile() in model_factory.py:

  • Defaults openai_chat_supports_multiple_system_messages=False, so leading system messages are merged (joined with \n\n). Harmless for endpoints that accept multiples — content is identical, just in one message.
  • Merges the existing thinking_tags profile and an optional extra= profile (preserves the Cerebras strict-tool-def override).
  • Per-model opt-out via "supports_multiple_system_messages": true (JSON boolean). A non-bool value raises TypeError instead of silently inverting intent — bool("false") is True in Python, so coercion can't save a string config.
  • Applied to custom_openai, openrouter, cerebras, zai_coding, and zai_api. Routing Cerebras through the shared helper also picks up thinking_tags, which the old bare profile skipped.
  • Plain openai/azure_openai are intentionally untouched: real OpenAI accepts multiple system messages, and merging would forfeit inter-system-message prompt-cache breakpoints.

README documents the default behavior, the SGLang symptom, and the opt-out.

Testing

  • Per-model-type profile assertions: custom_openai (default + opt-out), openrouter, cerebras (both flags), zai_coding, zai_api
  • Fail-fast guard: string "false" raises TypeError
  • Thinking-tags × extra × merge triple-merge, asserted against the runtime-resolved tags (non-vacuous)
  • End-to-end wire-format test: a post-compaction history with two leading system parts + instruction_parts passed through _map_messages yields exactly one merged system message
  • Full factory/basics/settings suites: 67 passed, 2 pre-existing env-gated skips; ruff format + ruff check clean

Fixes a 400 'System message must be at the beginning.' from SGLang /
vLLM-style endpoints when resuming after auto-compact. The wire format
ends up with two leading system messages -- the compaction summary
(SystemPromptPart produced by pydantic-ai-harness) plus the agent's
per-turn instruction_parts -- and strict backends accept only one.
pydantic-ai already ships _merge_leading_system_messages for exactly
this case, but it only fires when the model profile sets
openai_chat_supports_multiple_system_messages=False, which none of our
OpenAI-compatible model types did.

Add _strict_openai_profile() in model_factory.py:

- defaults openai_chat_supports_multiple_system_messages=False so
  leading system messages are merged (joined with \n\n); harmless on
  endpoints that accept multiples since content is identical
- merges any thinking_tags profile and an optional extra= profile
- per-model opt-out via "supports_multiple_system_messages": true
  (JSON boolean; non-bool values raise TypeError rather than silently
  inverting intent, since bool("false") is True)

Applied to custom_openai, openrouter, cerebras (via extra= alongside
its strict-tool-def override), and Zhipu zai_coding/zai_api. Routing
cerebras through the shared helper also picks up thinking_tags, which
the old bare profile skipped.

Tests: per-model-type profile flags, config override + fail-fast
guard, thinking-tags x extra x merge triple-merge, and an end-to-end
_map_messages wire-format test proving two leading system parts
collapse to exactly one merged system message. README documents the
default and the opt-out.
@thomwebb thomwebb self-assigned this Aug 23, 2026
@mpfaffenberger
mpfaffenberger merged commit 5ce787d into mpfaffenberger:main Aug 24, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants