What does that mean though, to “have access to our underlying thoughts”? Humans can obviously mentally do things that are impossible for a language model to do, because it’s trivial to show that humans do not need language to do mental tasks, and this includes things related to thought, so I don’t really get what is being argued in the first place.
> it’s trivial to show that humans do not need language to do mental tasks
LLMs don't need language to do mental tasks, either. Their input and output is language - like humans - but in between, the high-dimensional vector representations (often loosely called latent space) are not language in any meaningful sense.
LLMs can benefit from "thinking out loud" much as humans can. The issue is not whether the supposed "thoughts" are actually representative on any "internal" thoughts, but rather that explicating the problem in more detail can help reach better conclusions.
One point I was making is that the idea that humans are doing something "special" (or in the OP comment's terms, "real") in this area isn't well-supported, in fact there's plenty of evidence against it.
They're arguing that we have no evidence that humans have access to our underlying thoughts any more than the models do.
What does that mean though, to “have access to our underlying thoughts”? Humans can obviously mentally do things that are impossible for a language model to do, because it’s trivial to show that humans do not need language to do mental tasks, and this includes things related to thought, so I don’t really get what is being argued in the first place.
> it’s trivial to show that humans do not need language to do mental tasks
LLMs don't need language to do mental tasks, either. Their input and output is language - like humans - but in between, the high-dimensional vector representations (often loosely called latent space) are not language in any meaningful sense.
LLMs can benefit from "thinking out loud" much as humans can. The issue is not whether the supposed "thoughts" are actually representative on any "internal" thoughts, but rather that explicating the problem in more detail can help reach better conclusions.
One point I was making is that the idea that humans are doing something "special" (or in the OP comment's terms, "real") in this area isn't well-supported, in fact there's plenty of evidence against it.
4 replies →