Comment by storus
2 months ago
This is a tokenizer issue. Just ask it to first spell out the word letter by letter and then ask the same question. Not sure why people use it as a proof of LLMs being stupid. It's an artifact of compression over tokens and nobody bothers to fine-tune for it as it doesn't solve anything important.
>Just ask it to first spell out the word letter by letter and then ask the same question.
In TFA, it's demonstrated that this is false. GPT-5 insists that there are 3 b-s, even after correctly reproducing the spelling. Check the images in the linked bsky post.
If it can spell the word letter by letter when you ask it, it must know what letters are in the word and be able to count them.
That is not how LLMs work. I don't know either how they work - but what you describe is demonstrated in TFA. It can spell the word correctly every time, and at the same time it can get the letter count wrong.