Skip to content

fix(tokenizer): extend decoding fix to cover Mistral no-spaces bug#2285

Open
eeshsaxena wants to merge 1 commit into
Lightning-AI:mainfrom
eeshsaxena:fix/1822-mistral-decode-missing-spaces
Open

fix(tokenizer): extend decoding fix to cover Mistral no-spaces bug#2285
eeshsaxena wants to merge 1 commit into
Lightning-AI:mainfrom
eeshsaxena:fix/1822-mistral-decode-missing-spaces

Conversation

@eeshsaxena

Copy link
Copy Markdown

Fixes #1822

When chatting with Mistral models, the output was missing spaces between words. This is because Mistral uses the same SentencePiece-backed tokenizer as LLaMA (both have the prefix-space issue documented in huggingface/transformers#31643).

The existing apply_decoding_fix only checked for 'LlamaTokenizer' in the tokenizer_class, but Mistral checkpoints report 'MistralTokenizer' or 'MistralTokenizerFast', so the fix was never applied for them.

Changes:

  • Extended the condition to also check for 'MistralTokenizer' in the tokenizer_class string
  • Used .get() instead of direct key access to avoid KeyError if tokenizer_class is missing
  • Updated the comment to explain the root cause (SentencePiece prefix-space encoding)

File changed:

  • litgpt/tokenizer.py

@eeshsaxena

Copy link
Copy Markdown
Author

Closing to test the fix locally first and add a test case before resubmitting.

@eeshsaxena eeshsaxena closed this Jul 16, 2026
@eeshsaxena eeshsaxena reopened this Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

chatting with mistral generates answer with no spaces

1 participant