Comment by daishi55
6 hours ago
> out of 100 random questions I might think to ask, it’s likely to say something wrong or stupid a handful of times at least.
What are some examples?
6 hours ago
> out of 100 random questions I might think to ask, it’s likely to say something wrong or stupid a handful of times at least.
What are some examples?
There's a benchmark for this and a lot of models get negative scores because they're so unreliable: https://artificialanalysis.ai/evaluations/omniscience
I wanted some examples they actually experienced. Because I use these things daily and haven’t seen a hallucination in a long long time.
I saw a hallucination just this afternoon about a spurious ca cert error. Definitely happens less often, but I do need to correct it occasionally. Maybe once a week so it still requires vigilance.
Search a terminal with Claude Code for things like, “I got it wrong twice. I should look up the documentation instead of guessing.”
Does it about once a day, that I notice.