Comment by ben_w

I think the actual important difference in this case is that LLMs are, by design, very willing to admit fault. I suspect, but cannot yet prove, that this is because corrigibility (important part of AI alignment & safety research) has a significant vector similarity to fawning and to sycophancy.

With regard to them not, in any capacity whatsoever having a concept of factual correctness, LLMs are very much just like humans: We're not magic, we don't know the underlying nature of reality.

This is why it took us so long to replace Aristotelean physics with Newtonian, let alone Newtonian with QM and GR, and both QM and GR are known to be flawed but nobody has worked out the next step. It's just that humans are fairly unwilling to change their minds about how physics works in light of evidence, we often just defer to famous people, c.f. to Aristotle, then to Newton, then to Einstein.

We humans make this (opposite) mistake so hard and so often, that there's a saying that "science progresses one funeral at a time": https://en.wikipedia.org/wiki/Planck%27s_principle

I could also have thrown into this list miasma, phlogiston, that Columbus definitely wasn't the only person who knew it was round and just got lucky with the existence of the Americas after having gotten the size of both Earth and Asia catastrophically wrong, or phrenology.