Update tokenizer_config.json
#3
by
ArthurZ
HF Staff
- opened
This should make it compatible with v5 + corrects it because it does not follow gpt2.
GPT2 does not have a normalizer, and has a different pre-tokenizer https://huggingface.co/openai-community/gpt2/raw/main/tokenizer.json
awni
changed pull request status to
merged
