fix(tokenizer): look up the decoding-fix dummy token instead of hardcoding its id

[eeshsaxena]

Fixes #1822.

Tokenizer.decode works around the SentencePiece leading-space stripping by prepending a token that decodes to \x1e, then slicing that prefix back off. The id was probed as 33, falling back to 165 for salamandra. In the Mistral vocabularies neither is right - id 33 is [control_31] and 165 is [control_163], both of which decode to an empty string, so dummy_token is "", the slice is a no-op, and every single-token decode loses its leading space. That is the run-on output in the issue.

Repro against the real mistralai/Mistral-7B-Instruct-v0.3 tokenizer:

full decode      : "Hello, readers of this test!"
per-token decode : "Hello,readersofthistest!"    # before
per-token decode : " Hello, readers of this test!"  # after

The fix keeps the 33/165 fast path and otherwise looks \x1e up in the vocabulary (it is id 31918 for Mistral v0.3). Llama is unaffected - it still resolves to 33 and produces identical output, checked against TinyLlama-1.1B-Chat-v1.0.

test_tokenizer_against_hf had a if not config.name.startswith(("Mistral", "Mixtral")) guard around the per-token decode assertion, which is exactly the case that was broken, so this removes it.

I could not run the full test module locally (lightning is not installed on this machine), so I verified by loading litgpt/tokenizer.py directly against tokenizer assets downloaded from the two HF repos above rather than by running pytest. CI should cover the rest.

[eeshsaxena]

Closing to test the fix locally first and add a test case before resubmitting.

[eeshsaxena]

Ignore my earlier comment about closing this - I kept it open and reworked it instead, because the original diff was wrong. There is no MistralTokenizer class: Mistral's tokenizer_config.json already says "tokenizer_class": "LlamaTokenizer", so apply_decoding_fix was already True and that condition was dead code.

The real cause is the hardcoded dummy token id in decode. Details and the repro are in the updated description.