fix(tokenizer): look up the decoding-fix dummy token instead of hardcoding its id - #2285

fix(tokenizer): look up the decoding-fix dummy token instead of hardcoding its id#2285

eeshsaxena wants to merge 2 commits into

Lightning-AI:mainLightning-AI/litgpt:mainfrom

eeshsaxena:fix/1822-mistral-decode-missing-spaceseeshsaxena/litgpt:fix/1822-mistral-decode-missing-spacesCopy head branch name to clipboard