Comment by kadushka
4 days ago
These models are not trained on character level input. Why would anyone expect them to perform well on character level puzzles?
4 days ago
These models are not trained on character level input. Why would anyone expect them to perform well on character level puzzles?
emergent behavior. These things are surprisingly good at generalizing
They are trained on many billions of tokens of text dealing with character level input, they would be rather dumb if they couldn't learn it anyway.
Every human learns that, when you hear the sound "strawberry" you don't hear the double r there, yet you still know the answer.
These models operate on tokens, not characters. It’s true that training budgets could be spent on exhaustively enumerating how many of each letter are in every word in every language, but it’s just not useful enough to be worth it.
It’s more like asking a human for the Fourier components of how they pronounce “strawberry”. I mean the audio waves are right there, why don’t you know?
Although a vast majority of tokens are 4+ characters, you’re seriously saying that each individual character of the English alphabet didn’t make the cut? What about 0-9?
14 replies →