Comment by Al-Khwarizmi
1 year ago
An LLM doesn't even see individual letters at all, because they get encoded into tokens before they are passed as input to the model. It doesn't make much sense to require reasoning with things that aren't even in the input as a requisite for intelligence.
That would be like an alien race that could see in an extra dimension, or see the non-visible light spectrum, presenting us with problems that we cannot even see and saying that we don't have AGI when we fail to solve them.
And yet ChatGPT 3.5 can tell me the nth letter of an arbitrary word…
I have just tried and it indeed does get it right quite often, but if the word is rare (or made up) and the position is not one of the first, it often fails. And GPT-4 too.
I suppose if it can sort of do it is because of indirect deductions from training data.
I.e. maybe things like "the third letter of the word dog is d", or "the word d is composed of the letters d, o, g" are in the training data; and from there it can answer questions not only about "dog", but probably about words that have "dog" as their first subtoken.
Actually it's quite impressive that it can sort of do it taking into account that, as I mention, characters are just outright not in the input. It's ironic that people often use these things as an example of how "dumb" the system is when it's actually amazing that it can sometimes work around that limitation.
...because it knows that the next token in the sequence "the 5th letter in the word _illusion_ is" happens to be "s". Not because it decomposed the word into letters.
It seems unlikely that such sequences exist for the majority of words. And I asked in English about Portuguese words.
And yet GPT4 still can't reliably tell me if a word contains any given letter.