Comment by minimaxir

Yes, I know how tokenizers work and have spent an embarrassing amount of time working with/training tokenizer models with Hugging Face tokenizers.