I think we should go a little deeper on this idea.
We can all agree that both human "experts" and LLMs can sometimes be right, and sometimes be confidently wrong.
But that doesn't imply that they're equally fit for purpose. It just means that we can't use that simple shortcut to conclude that one is inferior to the other.
I’ve always thought of the definition of “expert” as reliably knowing the difference between what is known, what is speculated but unproven, and what is unknown. People claim expertise in all sorts of things that they aren’t experts in. But true experts should not be wrong. They should qualify levels of certainty. This definition certainly works in the sciences.
In reality few humans are true experts on every topic they open their mouth on. A high school teacher in science is hardly a true expert in every single thing they teach.
I think we should go a little deeper on this idea.
We can all agree that both human "experts" and LLMs can sometimes be right, and sometimes be confidently wrong.
But that doesn't imply that they're equally fit for purpose. It just means that we can't use that simple shortcut to conclude that one is inferior to the other.
So where do we go from here?
I’ve always thought of the definition of “expert” as reliably knowing the difference between what is known, what is speculated but unproven, and what is unknown. People claim expertise in all sorts of things that they aren’t experts in. But true experts should not be wrong. They should qualify levels of certainty. This definition certainly works in the sciences.
In reality few humans are true experts on every topic they open their mouth on. A high school teacher in science is hardly a true expert in every single thing they teach.