Comment by imtringued
1 year ago
This doesn't make any sense. The LLM knows the token to letter associations if you ask it. Once it has split the token into letters, it can answer the question anyway.
This tells us that the model is unaware of its own limitations, otherwise it would have figured that this tasks needs it to split the letters first.
Well splitting into letters doesn't help with Mistral Large 2. I tried this with Mistral 2:
[Question: How many r's in strawberry? First split the word into letters, one letter on each line then answer the question.]
[Answer: Let's split the word "strawberry" into letters:
s t r a w b e r r y
Now, let's count the number of 'r's:
There are 2 'r's in the word "strawberry".]
[Question: Did you count the number of r's?]
[answer: Yes, I did. Here's the count:
r r
So, there are 2 'r's in the word "strawberry".]
I suspect the issue is these models have to be trained to accomplish tasks, and this is not a task it makes sense to train a model to perform. You might train it to do this with function calling one day, calling a python library to count the r's in a string, but actually manually counting r's just doesn't play to an LLM's strengths.
Does it? It knows how a word in some context represented by tokens split into letters. It may know or not know the table. I wouldn’t trust what it tells about word/token correspondence more than in general.