Comment by zephen
2 hours ago
It is a well-known trope that LLMs are prediction engines.
But, to get the best out of them, you really have to consider what that means in the small. They are predicting, to a first order of approximation, what you want or expect them to answer.
Its response to your first prompt is hilarious, because the LLM completely misunderstood you and based its prediction on what it thought you wrote. Its response to your second prompt further cements that its goal is to predict what you want or expect to see.
It's also well known that LLMs are prone to hallucinations. One of the biggest triggers for hallucinations is when the LLM's interpretation of your expectations doesn't match reality.
Because the LLM will try to make reality match what it perceives to be your expectations.
One of the best ways to reduce hallucinations is to work hard to remove any assertions from your prompts.
For example "Isn’t it crazy that X is better than Y." contains an explicit assertion. The LLM misunderstood the direction of the assertion, but certainly understood that an assertion was there, and so it gave you reasons why reality matched its understanding of your assertion.
When you clarified the assertion, it switched, and again gave you reasons why reality matches its understanding of your assertion.
Lawyers often get into trouble for made-up citations. "Claude, find me case law that shows X" is a recipe for disaster. Instead "Claude, what is the case law on X?" is probably a better starting point.
Yeah I think this concerns me in two fronts and this specific tendency may be the line in the sand separating LLMs from AGI.
On the getting work done front, if what you’re trying to do is remotely subjective, than you really need to be sure you’re asking it the right thing and not expecting it to correct you and provide capital T truth.
On the social element. Like wow using it as a therapist or a mentor or to bounce ideas off of. What a huge trap if you expect it to correct you with a semblence of objective reality.
> What a huge trap if you expect it to correct you with a semblence of objective reality.
Yes, as a few cases have shown, people can go off the deep end, and the LLM goes right there along with them.
The LLM has no understanding of objective reality. It's even worse than any one of the blind men trying to describe an elephant, because it has no true experience of either the thing, or the other thing it is trying to compare it to.