Comment by moffkalast

It's more than just that, practically every notable open model released in the past year or so uses tiktoken as the tokenizer.