fix(llguidance): raise a clear error for CFG literals matching special tokens

[VenkateswarluNagineni]

Problem

Fixes #1771.

A CFG literal that exactly matches one of the tokenizer's special tokens (e.g. <think> on reasoning-model tokenizers like Qwen3-4B-Thinking) makes llguidance's parser fail with an opaque ParserTooComplex error at generation time:

Parser Error: token "<think>[151667]" doesn't satisfy the grammar; forced bytes: got '<'; applying 'ÿ'
Stop: ParserTooComplex

Root cause: the grammar's literal terminal ("<think>") is written for byte-level matching, but the tokenizer emits <think> as a single atomic token rather than as decomposable byte fragments. llguidance's parser can't reconcile an atomic token against a literal-string terminal, and bails out with a hard-to-diagnose error deep in generation.

What this does

This doesn't attempt to fix llguidance's parser itself (that lives in a separate compiled dependency, guidance-ai/llguidance, and reasoning-model support there looks like a bigger design effort — the maintainer's comment on the issue mentions this is being worked on). Instead, it detects the conflict at grammar-compile time, where we have both the grammar and the tokenizer's vocabulary available, and raises a clear ValueError naming the conflicting literal(s) instead of letting the confusing runtime parser failure surface later during generation.

Scoped to the Transformers backend (where the tokenizer's vocabulary is available) since that's what's reported and what I could verify. LlamaCpp and MLXLM tokenizers expose this differently and aren't covered here.

Update: the initial version of this fix only checked all_special_tokens, which misses tokens registered with special=False. Reasoning-model tokenizers commonly register <think>/</think> exactly this way (verified against the real Qwen/Qwen3-4B-Thinking-2507 tokenizer — <think> is absent from all_special_tokens) specifically so skip_special_tokens=True decoding doesn't strip them mid-generation, but the tokenizer still emits them atomically, bypassing ordinary BPE merging exactly like a true special token does. Without this, the guard would have silently let the original issue's exact grammar through to the same opaque ParserTooComplex failure it was meant to catch. The check now covers the union of all_special_tokens and every token in added_tokens_decoder, since the special flag only governs decode-time stripping, not which tokens bypass BPE merging.

Testing

• Reproduced the original issue locally: built the llguidance matcher directly from the real Qwen/Qwen3-4B-Thinking-2507 tokenizer (no model weights needed for the grammar-compile-time check — confirmed separately end-to-end with the actual model loaded too) and confirmed the exact ParserTooComplex failure from the issue, then confirmed the fix raises a clear ValueError instead.
• Added test_cfg_logits_processor_rejects_literal_matching_special_token, using the existing lightweight erwanf/gpt2-mini fixture already in this test file (its <|endoftext|> special token stands in for <think>, so the test doesn't need network access to a 4B+ model).
• Added test_cfg_logits_processor_allows_literal_not_matching_special_token to confirm ordinary grammars are unaffected.
• Added test_cfg_logits_processor_rejects_literal_matching_added_non_special_token, reproducing the added-but-not-special gap via tokenizer.add_tokens(["<think>"], special_tokens=False) on the same lightweight fixture — no network access to the 4B model needed here either.
• Ran the full test_llguidance_backend[model_transformers-torch] parametrization (the one exercising get_cfg_logits_processor with the existing cfg_lark/cfg_ebnf fixtures end-to-end through real generation) to confirm no regression.
• ruff check clean on both changed files (matches the pinned pre-commit ruff version, 0.9.1). pre-commit run mypy reports two pre-existing, unrelated urllib3 stub errors in src/outlines/exceptions.py present identically without this change (confirmed by re-running against the base commit) — not introduced by this PR.

[ErenAta16]

The diagnosis reads right to me, and the docstring explaining why added_tokens_decoder is included alongside all_special_tokens — that special only governs decode-time stripping, not whether the tokenizer splits the text out before BPE — is the part I'd have got wrong. Worth keeping.

The literal extraction is where I'd look again. Running the regex over a few grammar shapes:

grammar	literals found	conflict raised
start: "<think>" WORD	['<think>']	yes
start: '<think>' WORD	[]	no
start: "<think>" WORD	['\\u003cthink\\u003e']	no
// example: "<think>" + rule	['<think>']	yes, from a comment
start: "say\"hi" WORD	['say\\"hi']	no

Four things fall out of that.

Single-quoted literals are missed. Lark accepts '<think>' and "<think>" interchangeably, and the pattern only looks at double quotes. Someone hitting the original ParserTooComplex and switching quote style to see if it helps would get the opaque error back with no explanation. Adding '((?:[^'\\]|\\.)*)' as an alternation covers it.

Escapes aren't decoded before comparison. The capture keeps the raw text, so "<think>" and "say\"hi" are compared as their source form rather than the string they denote. Both are ways to write a conflicting literal that the check waves through. The escapes a grammar author would realistically use here are \" and \uXXXX, so decoding those two before the set intersection would close most of it.

A literal inside a comment raises. That's the direction that costs the user something — a valid grammar refuses to compile because a token name appears in a // comment. Stripping comments before the scan, or at least mentioning the limitation in the error, would avoid a confusing hard failure.

The check is exact-match only. "<think>abc" embeds the atomic token without equalling it, so it passes the check and presumably still confuses the parser. That may be deliberate scoping — matching substrings would produce false positives on any grammar containing < — but if so it's worth a line saying the check covers exact literals only, since the error message reads like a general guarantee.

None of these break the reported case; they're all ways the same bug reaches a user without the new error firing. Given the point of the PR is turning an opaque failure into an actionable one, the quote-style gap in particular seems worth closing before merge — it's the first thing someone debugging this would try.

[VenkateswarluNagineni]

Thanks for the close read — the regex table made the gaps concrete. Pushed 269a7f87 to address all three:

• Single-quoted literals: the scanner now matches '...' as well as "...".
• Undecoded escapes: literal text is now escape-decoded (\\, \", \', \n, \t, \r, \uXXXX) before comparing against token text, so "<think>" is caught same as "<think>".
• Literal inside a comment: rather than stripping comments as a separate pass, I combined literal and comment matching into one regex scan — a // that occurs inside a string (e.g. "http://foo") is consumed as part of that string because the string alternative starts matching at the opening quote, while a real // comment outside any string is matched by the comment branch and its contents are never extracted as literals. That also sidesteps the risk of a separate comment-stripping pass mangling a literal that legitimately contains //.

On exact-match-only: kept that scoping as-is (matching substrings would indeed produce false positives on grammars containing < etc.), but added a line to the docstring making clear the check only catches exact literal matches, not embedded occurrences.

Added regression tests for each of the four cases from your table (single-quote, \u-escape, comment false-positive, plus a same-shape check that a literal legitimately containing // still compiles). Full test_llguidance.py CFG-literal suite passes locally; ruff and mypy clean on the touched files.

[ErenAta16]

Re-ran the same table against 269a7f87. All five rows flip, and the //-inside-a-literal case the new comment mentions holds too:

grammar	literals found	conflict
start: "<think>" W	['<think>']	yes
start: '<think>' W	['<think>']	yes (was missed)
start: "<think>" W	['<think>']	yes (was missed)
// ornek: "<think>" + rule	[]	none (was a false positive)
start: "say\"hi" W	['say"hi']	yes (was missed)
start: "http://foo" W	['http://foo']	none

The alternation ordering is the part worth keeping the comment on. Putting the string alternatives before //[^\n]* means a // inside a literal gets consumed by the string match, so "http://foo" survives intact instead of having the rest of the line swallowed as a comment — and because only dq/sq are read, a literal written inside a real comment never reaches the set. Two problems solved by one ordering decision, which isn't obvious from reading the regex alone.

Decoding \uXXXX alongside the simple escapes covers the case I'd worry about most in practice, since a grammar author working around the original ParserTooComplex is fairly likely to try escaping the angle brackets.

Substring matching still isn't covered ("<think>abc" passes), but you've said so explicitly in the docstring now, which was the ask — the error message no longer reads like a guarantee it can't make.

[ErenAta16]

Ran the new scanner against the cases I was worried about rather than eyeballing it, and all three gaps are genuinely closed. Extracting with _LITERAL_OR_COMMENT_RE + _decode_literal_escapes at 269a7f87:

grammar	extracted
start: "<think>"	<think>
start: '<think>'	<think>
start: "\u003cthink\u003e"	<think>
// avoid "<think>" + start: "ok"	ok
start: "http://x" | "<think>"	http://x, <think>
// don't + start: "<think>"	<think>
// stray " + start: "<think>"	<think>
start: "don't"	don't
start: 'say "hi"'	say "hi"
start: "a\"b"	a"b

The two that could plausibly have gone wrong both hold up. An unpaired ' or " inside a comment doesn't desynchronise the scan, because the comment alternative is tried at the // before the scan ever reaches the stray quote. And "http://x" survives, since the string alternative starts matching at the opening quote and swallows the //. Combining the two into one pass was the right call — a separate comment-stripping pass is exactly where that class of bug lives.

Two observations, neither blocking:

Lark regexp terminals aren't distinguished from string literals. STR: /"[^"]*"/ extracts [^ as a literal. It's harmless in practice — a fragment like that can't equal a token string, so it can't produce a false error — but the extracted set isn't strictly "grammar literals", it's "quoted runs". Might be worth a word in the docstring so nobody later builds something stricter on top of it.

added_tokens_decoder is read unguarded in __init__. pyproject.toml doesn't pin transformers, and the attribute only exists from 4.34 onward. Because the call sits in the constructor rather than in get_cfg_logits_processor, a tokenizer without it takes down LLGuidanceBackend construction entirely, so JSON-schema and regex users would hit an AttributeError from a CFG-only feature. getattr(hf_tokenizer, "added_tokens_decoder", {}) would keep the blast radius on the code path that actually needs it.

The special=False reasoning in the docstring is the part I'd most want kept — that added-but-not-special tokens still bypass BPE is the non-obvious fact that makes this check correct, and it's the sort of thing that gets "simplified" away later without it written down.

[VenkateswarluNagineni]

Pushed bf148779 addressing both follow-ups:

1. added_tokens_decoder guard: changed to getattr(hf_tokenizer, "added_tokens_decoder", {}) — falls back to an empty dict on transformers < 4.34, so the atomic-token set degrades to all_special_tokens only rather than raising AttributeError for JSON-schema and regex users.

2. Regexp terminal note: added a sentence to the _extract_grammar_literals docstring acknowledging that the scanner matches quoted runs without regard to surrounding Lark syntax, so content like the "[\^" fragment inside STR: /"[\^"]*"/ may be extracted — but can't equal any real token string and can't produce a false positive.

The special/added token distinction in the docstring is unchanged, as requested.