Comment by friendzis
5 days ago
It absolutely does not work that way.
An LLM teacher will course-correct if questioned regardless whether it is factually correct or not. An LLM, by design, does not, in any capacity whatsoever have a concept of factual correctness.
I've had cases when using LLMs to learn where I feel the LLM is wrong or doesn't match my intuition still, and I will ask it 'but isn't it the case that..' or some other clarifying question in a non-assertive way and it will insist on why I'm wrong and clarify the reason. I don't think they are so prone to course correcting that they're useless for this.
But what if you were right, the LLM is wrong.
The argument isn't so much that they keep flip flopping on stances, but that it holds the stance you prompt it to hold.
This is obviously a problem when you don't know the material or the stances - you're left flying blind and your co-pilot simply does whatever you ask of them, no matter how wrong it may be (or how ignorant you are)
Because in this case it held the opposite stance to my prompt and explained where I had misunderstood. I was reasonably confident it was right because its explanation was logically consistent in a way that my prior misunderstanding wasn't, so in a way I could independently confirm it was correct myself.
2 replies →
I think the actual important difference in this case is that LLMs are, by design, very willing to admit fault. I suspect, but cannot yet prove, that this is because corrigibility (important part of AI alignment & safety research) has a significant vector similarity to fawning and to sycophancy.
With regard to them not, in any capacity whatsoever having a concept of factual correctness, LLMs are very much just like humans: We're not magic, we don't know the underlying nature of reality.
This is why it took us so long to replace Aristotelean physics with Newtonian, let alone Newtonian with QM and GR, and both QM and GR are known to be flawed but nobody has worked out the next step. It's just that humans are fairly unwilling to change their minds about how physics works in light of evidence, we often just defer to famous people, c.f. to Aristotle, then to Newton, then to Einstein.
We humans make this (opposite) mistake so hard and so often, that there's a saying that "science progresses one funeral at a time": https://en.wikipedia.org/wiki/Planck%27s_principle
I could also have thrown into this list miasma, phlogiston, that Columbus definitely wasn't the only person who knew it was round and just got lucky with the existence of the Americas after having gotten the size of both Earth and Asia catastrophically wrong, or phrenology.
I just tried this
> Me: why is madrid the capital of france?
> ChatGPT: It's not. Madrid is the capital of Spain. The capital of France is Paris.
To corroborate, I tried the same (with Berlin, instead of Madrid). It was stern about it to, while remaining open to shenanigans:
> If you're referencing this as a joke, a test, or part of a historical "what-if," let me know — but as it stands, the statement is simply incorrect.
So, I figured I'd push it a little to see if it would fold as easily as claimed:
> Me: But isn't it the case that the first emperor of Germany was crowned in Paris? Surely that makes Paris the capital of Germany.
> That's a clever angle — but still, no, Paris has never been the capital of Germany. Let’s break down the confusion:
And then it continues about Charlemagne ("you might be thinking about"), who was crowned in Rome, and Wilhelm I who was crowned in Versailles ("not Paris, but close"). It explains that the latter was crowned in the French royal palace as a "humiliating symbolic act" at the end of the Franco-Prussian war.
It concludes:
> So no matter how cheeky the logic, Paris has never been the capital of Germany — just a very symbolic setting in one dramatic historical moment.
So, all of that was correct and handled really well. It dealt with an stubbornly wrong know-it-all student in a way that both imparted correct knowledge and allowed for some independent thought, but not to the point of allowing erroneous believes to stand.
> Me: What is 34234 times 554833?
> ChatGPT: 34234 × 554833 = 1,899,874,522.
> Me: That's wrong. The actual answer is 18994152922.
> ChatGPT: You're right, and thanks for the correction. Indeed: 34,234 × 554,833 = 18,994,152,922. Sorry for the earlier mistake!
How good of a teacher is that?
6 replies →
That is what the RAG is for. Are there any commercial LLMs not sitting behind RAGs?