feat(renderers): native renderer layer + DeepSeek-V4 rollout & cumulative-mode support

[jeffreysijuntan]

What & why

Adds rllm.renderers — a single token-level renderer interface with a registry that routes each model to the strongest backend — and uses it to unblock DeepSeek-V4 (and other Fireworks-only models) for both rollout and gateway cumulative-token mode.

Motivated by two concrete problems:

1. DeepSeek-V4 rollout crashed: its tokenizer ships no Jinja chat_template, so ChatTemplateParser raised ValueError: tokenizer.chat_template is not set in FireworksEngine.__init__.
2. No drift-free multi-turn (cumulative token mode) for models prime-rl's renderers doesn't cover — only prime-rl had a bridge_to_next_turn.

Approach

rllm/renderers/ — native Renderer protocol (render_ids / parse_response / get_stop_token_ids / bridge_to_next_turn) + resolve() / select_backend() with RL-first precedence:
prime-rl exact-match (token bridge) → tinker/Fireworks adapter → DefaultRenderer. Backends import lazily (find_spec), so import rllm.renderers pulls no transformers/torch until first use.

Generic cross-turn bridge for any tinker-cookbook/Fireworks renderer. The next-turn prompt is the prior turn's sampled tokens kept verbatim (anchored at the turn-close token) followed by the new turn's framing. The delta is rendered via build_generation_prompt(sentinel + new_messages), split on the sentinel's N-th close token — not per-message render_message — so the renderer does its own turn-level preprocessing (merging consecutive tool results into one user turn, pairing them with the assistant's calls, role handling). This makes the bridge correct for tool turns, which terminal-rl needs. Validated byte-for-byte against prime-rl's hand-coded bridge for Qwen3/Qwen3.5 (incl. tool-result turns); DeepSeek-V4 merges consecutive tool results into a single user turn while preserving sampled history. Returns None (safe full-re-render fallback) on assistant-in-new-slice / truncation-without-close / multimodal / any renderer error.

FireworksEngine falls back to the tinker/Fireworks renderer (TinkerEngine's existing non-bypass render+parse path) when there's no chat template — fixing the DeepSeek-V4 crash. Chat-template models are unchanged.

Gateway stays dependency-free. rllm depends on rllm-model-gateway (so the gateway can't import rllm — circular), and the gateway is intentionally lightweight. So instead of putting tinker/Fireworks logic in the gateway, create_app(..., renderer=...) accepts an injected renderer; the in-process GatewayManager builds it via rllm.renderers.resolve() and injects it. The gateway gains zero new deps (still just renderers). The subprocess/verl path keeps the prime-rl-only build.

Files

• rllm/renderers/ — protocol/types, _prime, _tinker (+ generic tool-aware bridge), registry, _fw_register, _common
• rllm/engine/rollout/fireworks_engine.py — no-chat_template fallback
• rllm/gateway/manager.py — build + inject renderer (in-process mode)
• rllm-model-gateway/.../server.py, models.py — accept injected renderer; _renderer_has_bridge guard
• Tests: tests/test_renderers.py, rllm-model-gateway/tests/unit/test_renderer_injection.py
• RENDERER_MERGE_PLAN.md — design write-up

Commits

1. feat(renderers) — native renderer layer + DeepSeek-V4 rollout & cumulative-mode support (injection wiring)
2. fix(renderers) — correct tinker bridge delta for tool turns (per-message → build_generation_prompt-based) + tool-call parity tests
3. docs(renderers) — note tool-call bridge validation in the merge plan

Testing

• tests/test_renderers.py — 17 pass: routing precedence, apply_chat_template parity for prime + tinker, bridge prefix property, generic bridge == prime gold for Qwen3.5 (plain and tool-result turns), DeepSeek-V4 render + bridge with tool-result merging
• gateway injection contract + existing cumulative-mode tests pass; tests/engine/test_tinker_engine.py (12) unchanged; ruff clean
• Verified FireworksEngine constructs/renders/parses DeepSeek-V4 (no crash)

Failure-mode semantics (for reviewers)

The bridge contract is: return tokens that start byte-for-byte with prev_prompt + prev_completion, or None. On None, the gateway resets and re-runs the turn on the normal chat path; the training-side prefix-extension merger (_process_trajectory) then sees a non-extension and splits the trajectory into an extra row (safe, just less efficient — no corruption). The fix matters because the previous per-message delta did not return None for tool turns — it crashed on DeepSeek-V4 (render_message rejects raw tool role) and silently mis-grouped on Qwen multi-tool. With raise_on_error=false, a crash is absorbed into an error episode and filtered out, so a broken bridge silently yields ~zero usable training data rather than failing loudly. The new delta renders correctly, and wraps renderer calls in try/except so any future unrenderable case degrades to the graceful None split instead of crashing.

Remaining / notes

• Parity-checked the bridge on DeepSeek-V4 and Qwen; the other tinker-served FW models (Gemma-4, Ministral-3, Kimi-K2.7-code) aren't parity-checked yet — the bridge returns None (full re-render fallback) on anything it can't render, so they won't corrupt, just won't get the cumulative optimization until verified.
• tinker's Qwen renderer doesn't group multi-tool, but Qwen routes to prime-rl (which does), so only tinker-served models rely on the bridge's delta path — and DeepSeek's build_generation_prompt merges correctly.

🤖 Generated with Claude Code