Comment by gwervc
1 year ago
> But then I started asking myself how can we figure out the limits of its ability to reason
Third paragraph. The entire article is based on the premise LLMs are supposed to reason, which is wrong. They don't, they're tools to generate text.
I really hate this reductive, facile, "um akshually" take. If the text that the text-generating tool generates contains reasoning, then the text generation tool can be said to be reasoning, can't it.
That's like saying "humans aren't supposed to reason, they're supposed to make sounds with their mouths".
At some point if you need to generate better text you need to start creating a model of how the world works along with some amount of reasoning. The "it's just a token generator" argument fails to get this part. That being said I don't think just scaling LLMs are going to get us AGI but I don't have any real arguments to support that
> If the text that the text-generating tool generates contains reasoning, then the text generation tool can be said to be reasoning, can't it.
I don't know... you're still describing a talking parrot here, if you'd ask me.
I’m not a fan of the talking parrot argument, especially when you’re pointing it at models of scale.
The only thing separating a talking parrot and humans is our accuracy in shaping our words to the context in which they’re spoken.
Sure it’s easy to liken a low resource model to a talking parrot, the output seems no better than selective repetition of training data. But is that really so different from a baby whose first words are mimics from the environment around them?
I would argue that as we learn language we implicitly develop the neural circuitry to continue to improve our lexical outputs, this circuitry being concepts like foresight, reasoning, emotion, logic, etc and that while we can take explicit action to teach these ideas, they naturally develop in isolation as well.
I don’t think language models, especially at scale, are much different. They would seem to similarly acquire implicit circuitry like we do as they are exposed to more data. As I see it, the main difference in what exactly that circuitry accomplishes and looks like in final output has more to do with the limited styles of data we can provide and the limitations of fine tuning we can apply on top.
Humans would seem to share a lot in common with talking parrots, we just have a lot more capable hardware to select what we repeat.
1 reply →
What's the difference between a human and a talking parrot that can answer any question you ask it?
4 replies →
I feel the use of the word "parrot" is unintentionally apt, given that parrots were long thought to be mere mimics but were ultimately shown to have (at least the capacity for) real linguistic understanding.
Even if the generated text contains reasoning, could the LLM understand and apply it?
If I tell GPT-4 to print something, it understands it needs to check if my printer is turned on first and turn it on if it's not, so, yes?
Also, if the generated text contains reasoning, what's your definition of "understanding"? Is it "must be made of the same stuff brains are"?
5 replies →