Comment by fluoridation
2 months ago
>There's still a gap here between "has some hidden internal state" and "that state can provide insight into to their thought process".
No, because that internal state is part of the thought process. That's the whole point. You ask the human a question to learn something that you don't already know. It makes no sense to ask an LLM that because it knows nothing you don't already know; you and the LLM are privy to the exact same information. What's tripping you up about this?
>If we knew that asking this kind of question of a human was more likely to provide insights that improved the process in the future than asking it of an LLM, that would be interesting.
So, at this point I must ask: are you an NPC? Do you go through life just reacting to stimuli like a cockroach, with no understanding of why or how you do anything? If you're playing chess and someone asks you about a move you just made you are unable to explain, "I noticed such-and-such so I decided the best course of action was so-and-so to prevent this-and-that"? This is an alien concept to you? If so, then I'm sorry; most of us do not experience our own cognition in this way. We can perceive the formation of our own thoughts as well as the progressive retrieval of information.
>Meaning that a plausible explanation is valuable regardless of whether it's true? Wouldn't that apply just as well to an LLM's explanation?
See first paragraph.
> There's no question about whether other people have internal states. I can show someone a piece of information in such a way that only they see it and then ask them to prove that they know it such that I can be certain to an arbitrarily high degree that their report is correct.
> No, because that internal state is part of the thought process. That's the whole point. You ask the human a question to learn something that you don't already know.
If the internal state is entangled enough with in the thought process that it would help with providing insights, sure. But I don't know that humans have such state accessible to them, and the fact that humans can know facts that are not accessible from outside does not in itself convince me of that.
> It makes no sense to ask an LLM that because it knows nothing you don't already know; you and the LLM are privy to the exact same information.
OK but why does that mean that the LLM's explanation should be bad/useless, if the only difference is that I have more direct access to the LLM's information than I would to a human's information?
> So, at this point I must ask: are you an NPC? Do you go through life just reacting to stimuli like a cockroach, with no understanding of why or how you do anything? If you're playing chess and someone asks you about a move you just made you are unable to explain, "I noticed such-and-such so I decided the best course of action was so-and-so to prevent this-and-that"?
I can tell stories about my own cognition. Those stories feel real to me. But I'm aware that the best available scientific evidence suggests that they're indistinguishable from confabulations.
[dead]