Comment by furyofantares
1 year ago
It's actually that it has no idea how words are spelled because they're blind to letters. This is because they operate instead on tokens instead.
They do seem to know the first letter of each word pretty well (they love to alliterate and can also produce a list of things starting with A then B then C etc) but they are all very terrible at spelling beyond that. I presume they could be trained to spell even while retaining tokens but I guess I don't know for certain.
With tasks like these it helps if you break your words up with some kind of separator so that it all tokenizes to one character per token. They do know what individual letters are conceptually.
It helps a little. ChatGPT4 still fails the following prompt fairly often (maybe 40% of the time):
which of these 5-letter sequences a has P in position 4?
P U P I L
S T A P H
C R E E P
P O O P Y
L I P I D
It usually gets it if it reiterates each sequence before deciding (but not always) and it almost always fails when it just answers right away.
And this doesn't actually help at all with the actual related Wordle problem - "think of a 5-letter word with P in the fourth position".
You could try filling up its context with a list of a thousand of the most common 5 letter words all spelled out (and I've done so, even with additional hints like the positions of the letters in parens). But it really didn't help (as might be expected since it can't even reliably do it with the above list of 5 words.)