Comment by samus

2 months ago

It would make sense to repeat the experiment with the tokenizer of an LLM.

1 comment

samus

Reply

sebastianmestre 2 months ago

Pretty sure that OpenAI uses BPE in their GPT models