Comment by sigmoid10
1 day ago
These are usually from a missing context (i.e. specifying where the car is right now). Tokenizer issues (how many Xs are in Y) are a better giveaway, but models can solve them using tools nowadays. I think the best way to uncover an LLM is to reveal its superhuman aspects. Ask it for an A* algorithm for your problem. If it can just type down a working algorithm in a few seconds, that is definitely not a human. Although with some careful system prompt design, you could probably mask that aspect too.
> These are usually from a missing context (i.e. specifying where the car is right now).
That shouldn't be an issue. The very act of taking the car to car wash contains the information that it is with you and you are at your home. Doesn't this prove that LLMs don't actually think?
> These are usually from a missing context (i.e. specifying where the car is right now).
The reason for the mistake is irrelevant here. All that matters is that you catch the agent saying something that basically no human would, especially in this type of game.
If it is missing context, it will be indistinguishable from mistakes that humans could make as well.
No, the context is obvious to any human, that's the whole point. No one would think you've left your car at a car wash, as no one ever does that, and if they did, they wouldn't even consider driving there. It's obvious to any human that the question is a little trick and that the answer is extremely obvious.
1 reply →