Comment by encyclopedism
2 days ago
I couldn't agree with you more.
I really do find it puzzling so many on HN are convinced LLM's reason or think and continue to entertain this line of reasoning. At the same time also somehow knowing what precisely the brain/mind does and constantly using CS language to provide correspondences where there are none. The simplest example being that LLM's somehow function in a similar fashion to human brains. They categorically do not. I do not have most all of human literary output in my head and yet I can coherently write this sentence.
As I'm on the subject LLM's don't hallucinate. They output text and when that text is measured and judged by a human to be 'correct' then it is. LLM's 'hallucinate' because that is literally what they can ONLY do, provide some output given some input. They don't actually understand anything about what they output. It's just text.
My paper and pen version of the latest LLM (quite a large bit of paper and certainly a lot of ink I might add) will do the same thing as the latest SOTA LLM. It's just an algorithm.
I am surprised so many in the HN community have so quickly taken to assuming as fact that LLM's think or reason. Even anthropomorphising LLM's to this end.
Most of things that were considered reasoning are now trivially implemented by computers - from arithmetic, through logical inference (surely this is reasoning - isn't it) to playing chess. Now LLMs go even further - what is your definition of reasoning? What concrete action is in that definition that you are sure computer will not do in lets say 5 years?
The definition of things such as reasoning, understanding, intellect are STILL open academic questions. Quite literally humans greatest minds are currently attempting to tease out definitions, whatever we currently have falls short. For example see the hard problem of consciousness.
However I can attempt to provide insight by taking the opposite approach here. For instance what is NOT reasoning. Getting a computer to follow a series of steps (an algorithm) is NOT reasoning. A chess computer is NOT reasoning it is following a series of steps. The implications of assuming that the chess computer IS reasoning would have profound affects on so much, for example it would imply your digital thermostat also reasons!
> The simplest example being that LLM's somehow function in a similar fashion to human brains. They categorically do not. I do not have most all of human literary output in my head and yet I can coherently write this sentence.
The ratio of cognition to knowledge is much higher in humans that LLMs. That is for sure. It is improving in LLMs, particularly small distillations of large models.
A lot of where the discussion gets hung up on is just words. I just used "knowledge" to mean ability to recall and recite a wide range of fasts. And "cognition" to mean the ability to generalize, notice novel patterns and execute algorithms.
> They don't actually understand anything about what they output. It's just text.
In the case of number multiplication, a bunch of papers have shown that the correct algorithm for the first and last digits of the number are embedded into the model weights. I think that counts as "understanding"; most humans I have talked to do not have that understanding of numbers.
> It's just an algorithm.
> I am surprised so many in the HN community have so quickly taken to assuming as fact that LLM's think or reason. Even anthropomorphising LLM's to this end.
I don't think something being an algorithm means it can't reason, know or understand. I can come up with perfectly rigorous definitions of those words that wouldn't be objectionable to almost anyone from 2010, but would be passed by current LLMs.
I have found anthropomorphizing LLMs to be a reasonably practical way to leverage the human skill of empathy to predict LLM performance. Treating them solely as text predictors doesn't offer any similar prediction; it is simply too complex to fit into a human mind. Paying a lot of attention to benchmarks, papers, and personal experimentation can give you enough data to make predictions from data, but it is limited to current models, is a lot of work, and isn't much more accurate than anthropomorphization.
> The ratio of cognition to knowledge is much higher in humans that LLMs. That is for sure. It is improving in LLMs, particularly small distillations of large models.
It isn't a case of ratio it is a fundamentally different method of working hence my point of not needing all human literary output do the the equivalent of an LLM. Consider even the case of a person born blind they have an even more severe deficiency of input yet they are equivalent in cognitive capacity to a sighted person and certainly any LLM.
> In the case of number multiplication, a bunch of papers have shown that the correct algorithm for the first and last digits of the number are embedded into the model weights. I think that counts as "understanding";
Why are those numbers in the model weights? What if the model was trained on birdsong instead of humanities output would it then be able to multiply? Humans provide the connections, the reasoning the thought the insights and the subsequent correlations THEN we humans try to make a good pattern matcher/ guesser (the LLM) to match those. We tweak it so it matches patterns more and more closely.
> most humans I have talked to do not have that understanding of numbers.
This common retort: most humans also makes mistakes, or most humans also do x, y, z means nothing. Take the opposite implication of such retorts. For example most humans can't multiply 10 digits numbers therefore most calculators 'understand' maths better than most humans.
> I don't think something being an algorithm means it can't reason, know or understand. I can come up with perfectly rigorous definitions of those words that wouldn't be objectionable to almost anyone from 2010, but would be passed by current LLMs.
My digital thermometer uses an algorithm to determine the temperature. It does NOT reason when doing so. An algorithm is a series of steps. You can write them on a piece of paper. The paper will not be thinking if that is done.
> I have found anthropomorphizing LLMs to be a reasonably practical way to....
I think anthropomorphising is letting people assume they are more than they are (next token generators). In fact at the extreme end this anthropomorphising has led to exacerbating mental health conditions and unfortunately has even led to humans killing themselves.
You did not actually address the core of my points at all.
> It isn't a case of ratio it is a fundamentally different method of working hence my point of not needing all human literary output do the the equivalent of an LLM.
You can make ratios of anything. I agree that human cognition is different than LLM cognition, though I would think of it more like a phase difference than fundamentally different phenomena. Think liquid water vs steam, the density (a ratio) is vastly different and they have different harder to describe properties (surface tension, filling volume, incompressible vs compressible).
> Humans provide the connections, the reasoning the thought the insights and the subsequent correlations THEN we humans try to make a good pattern matcher/ guesser (the LLM) to match those.
Yes, humans provide the training data and benchmarks for measuring LLM improvement. Somehow meaning about the world has to get trained on to have any understanding. However, humans talking about patterns in number is not how the LLMs learned this. It is very much from just seeing lots of examples and deducing (during training not inference) the pattern. The fact that a general pattern is embedded in the weights implies that some general understand of many things are baked into the model.
> This common retort: most humans also makes mistakes, or most humans also do x, y, z means nothing.
It is not a retort, but some argument towards what "understanding" means. From what you have said, my guess of your definition makes "understanding" what humans do and computers are incapable of (by definition). If LLMs could out compete humans in all professional tasks, I think it would be hard to say they understand nothing. Humans are a worthwhile point of comparison and human exceptionalism can only really hold up until being surpassed.
I would also point out that some humans DO understand the properties of numbers I was referring to. In fact, I figured it out in second grade while doing lots of extra multiplication problems as punishment for being a brat.
> My digital thermometer uses an algorithm to determine the temperature. ... The paper will not be thinking if that is done.
I did not say "All algorithms are thinking". The stronger version of what I was saying is "Some algorithms can think." You simply have asserted the opposite with no reasoning.
> In fact at the extreme end this anthropomorphising has led to exacerbating mental health conditions and unfortunately has even led to humans killing themselves.
I do concede that anthropomorphizing can be problematic, especially if you do not have a background in CS and ML to understand beneath the hood. However, you completely skipped past my rather specific explanation of how it can be useful. On HN in particular, I do expect people to bring enough technical understanding to the table to not just treat LLMs as people.
I have had conversations at work, with people who I have reason to believe are smart and critical, in which they made the claim that humans and AI basically learn in the same way. My response to them, as to anyone that makes this claim, is that the amount of data ingested by someone with severe sensory dysfunction of one sort or another is very small. Helen Keller is the obvious extreme example, but even a person who is simply blind is limited to the bandwidth of their hearing.
And yet, nobody would argue that a blind person is any less intelligent that a sighted person. And so the amount of data a human ingests is not correlated with intelligence. Intelligence is something else.
When LLMs were first proposed as useful tools for examining data and proving answers to questions, I wondered to myself how they would solve the problem of there being no a-priori knowledge of truth in the models. How they would find a way of sifting their terabytes of training data so that the models learnt only true things.
Imagine my surprise that not only did they not attempt to do this, but most people did not appear to understand that this was a fundamental and unsolvable problem at the heart of every LLM that exists anywhere. That LLMs, without this knowledge, are just random answer generators. Many, many years ago I wrote a fun little Markov-chain generator I called "Talkback", that you could feed a short story to and then have a chat with. It enjoyed brief popularity at the University I attended, you could ask it questions and it would sort-of answer. Nobody, least of all myself, imagined that the essential unachievable idea - "feed in enough text and it'll become human" - would actually be a real idea in real people's heads.
This part of your answer though;
"My paper and pen version of the latest LLM .... My paper and pen version of the latest LLM"
Is just a variation of the Chinese Room argument, and I don't think it holds water by itself. It's not that it's just an algorithm, it's that learning anything usefully correct from the entire corpus of human literary output by itself is fundamentally impossible.
I concur with your sentiments.
> My paper and pen version of the latest LLM
My point here was to attempt to remove the mystery of LLM's by showing the same thing can be done with pen and paper version, after all an LLM is an algorithm. Because an LLM is running on a 'supercomputer' or is digital doesn't provide it some mysterious new powers.
People believe that because they are financially invested in it. Everyone has known LLMs are bullshit for years now.