Remove dead is_llama detection code in TransformerTokenizer#1876
Open
joaquinhuigomez wants to merge 1 commit into
Open
Remove dead is_llama detection code in TransformerTokenizer#1876joaquinhuigomez wants to merge 1 commit into
joaquinhuigomez wants to merge 1 commit into
Conversation
convert_token_to_string applies the SPIECE_UNDERLINE / <0x20> space workaround unconditionally, so the is_llama flag and the get_llama_tokenizer_types() helper that computed it are no longer read anywhere. Remove both, along with the test that exercised the helper, and drop the now-meaningless is_llama assignments in the conversion test. Fixes dottxt-ai#1874
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
convert_token_to_stringapplies theSPIECE_UNDERLINE/<0x20>space workaround unconditionally, so theis_llamaflag and theget_llama_tokenizer_types()helper that computed it are no longer read anywhere. This removes both, deletes the test that exercised the helper, and drops the now-meaninglessis_llamaassignments intest_transformer_tokenizer_convert_token_to_string(which still passes). Fixes #1874.