Comment by lmm
2 months ago
> Humans at least have a mental state that only they are privy to to work from
Maybe. How do you tell? What would you expect to be different if they didn't?
> The LLM literally cannot possibly have a deeper insight into the root cause than the user, because it can only work from the information that the user has access to.
Insight is not solely a function of available input information. Arguably being able to search and extract the relevant parts is a far more important part of having insights.
>Maybe. How do you tell? What would you expect to be different if they didn't?
I think you're asking how I would know if other people were P-zombies. That's an inappropriate question because I didn't talk about subjective experience, just about internal state. There's no question about whether other people have internal states. I can show someone a piece of information in such a way that only they see it and then ask them to prove that they know it such that I can be certain to an arbitrarily high degree that their report is correct.
Unvoiced thoughts are trickier to prove, but quite often they leave their mark in the person's voiced thoughts.
>Insight is not solely a function of available input information. Arguably being able to search and extract the relevant parts is a far more important part of having insights.
LLMs are notoriously bad at judging relevance. I've noticed quite often if you ask a somewhat vague question they try to cold-read you by throwing various guesses to see which one you latch onto. They're very bad at interpreting novel metaphors, for example.
> I didn't talk about subjective experience, just about internal state. There's no question about whether other people have internal states. I can show someone a piece of information in such a way that only they see it and then ask them to prove that they know it such that I can be certain to an arbitrarily high degree that their report is correct.
Well, sure, but that much is equally true for an LLM with a scratchpad or what have you. (I guess you could say that the user should have access to the LLM's scratchpad and therefore be just as able to understand the state as the LLM itself, but as we move towards the LLM using its own state vectors that's less and less true in practice). I agree that a human may have a mood or secret knowledge or what have you in a way that an LLM wouldn't, but if all you're positing is access to some inert but hidden state then that feels like a Toaster-Enhanced Turing Machine.
>if all you're positing is access to some inert but hidden state then that feels like a Toaster-Enhanced Turing Machine
I thought it was pretty clear, given the context. What I'm saying is that humans are capable of limited introspection in ways that LLMs are not. They can remember their thought processes and review them ex post facto to answer questions that LLMs cannot. An LLM fundamentally cannot truthfully answer questions such as "why did you do this?" because its entire working memory is held in the context window. It doesn't know to any greater degree than you because it has no more information than you do; just like they are for you, its internal workings are a mystery. I'm not saying LLMs conceptually could not be designed with capabilities similar to a human's in this regard, with some symbolic memory that's capable of some bookkeeping, I'm saying none of the current ones have them.
6 replies →