Comment by simonw
5 months ago
I hadn't realized OpenAI's tiktoken Python library could work with other models outside of the OpenAI family, that's really useful: https://github.com/therealoliver/Deepdive-llama3-from-scratc...
5 months ago
I hadn't realized OpenAI's tiktoken Python library could work with other models outside of the OpenAI family, that's really useful: https://github.com/therealoliver/Deepdive-llama3-from-scratc...
It's more than just that, practically every notable open model released in the past year or so uses tiktoken as the tokenizer.
I'm glad to have helped you :)