Comment by fc417fc802
2 months ago
Those are just words inside arbitrary tags, they aren't actually thoughts. Think of it as asking the model to role play a human narrating his internal thought process. The exercise improves performance and can aid in human understanding of the final output but it isn't real.
Why do you believe that humans have access to an “internal thought process”? I.e. what do you think is different about an agent’s narration of a thought process vs. a human’s?
I suspect you’re making assumptions that don’t hold up to scrutiny.
I made no such claim and I don't understand what direct relevance you believe the human thought process has to the issue at hand.
You appear to be defaulting to the assumption that LLMs and humans have comparable thought processes. I don't think it's on me to provide evidence to the contrary but rather on you to provide evidence for such a seemingly extraordinary position.
For an example of a difference, consider that inserting arbitrary placeholder tokens into the output stream improves the quality of the final result. I don't know about you but if I simply repeat "banana banana banana" to myself my output quality doesn't magically increase.
> I don't understand what direct relevance you believe the human thought process has to the issue at hand.
You're the one who raised it. Perhaps you should clarify what you mean by "isn't real" - do you believe a human narrating their thought process is saying something that's more real?
Someone else replied to your comment asking essentially the same question, perhaps better phrased:
> What would be different if it was "real"? What makes you think that when humans "narrate" "their" "internal thought process", it's any more "real"?
1 reply →
Given that LLMs can speak basically any language and answer almost any arbitrary question much like a human would, the claim that LLMs have comparable (not identical) thought processes to humans does not seem extraordinary at all.
Are you legitimately arguing that humans don’t have an internal thought process in some way?
They're arguing that we have no evidence that humans have access to our underlying thoughts any more than the models do.
6 replies →
What would be different if it was "real"? What makes you think that when humans "narrate" "their" "internal thought process", it's any more "real"?
I ask a human "predict what a mouse would do here". In an effort to understand why the prediction is sometimes wrong I ask "walk me through what the imaginary mouse is thinking". Upon examination I exclaim "aha! there's the error" but sadly it's not actually because the output prediction was not based on the thinking trace in any robust manner.
That's a loose analogy but it fails to fully illustrate the degree of decoupling here. For example the weirdness of LLM performance being increased via the output of empty sequences.
> I ask a human "predict what a mouse would do here". In an effort to understand why the prediction is sometimes wrong I ask "walk me through what the imaginary mouse is thinking". Upon examination I exclaim "aha! there's the error" but sadly it's not actually because the output prediction was not based on the thinking trace in any robust manner.
Is this meant to be an analogy for a human or an LLM? Where would it be different in the other case?