Comment by valleyer
2 days ago
Incidentally, according to https://gpt-tokenizer.dev, in gpt-5, "literature" is two tokens ("liter" + "ature"), whereas "文学" is one.
2 days ago
Incidentally, according to https://gpt-tokenizer.dev, in gpt-5, "literature" is two tokens ("liter" + "ature"), whereas "文学" is one.
My bad, the English word I had in mind was "technology" (技術) but I misremembered due to feeling like "tech" should be its own token.