Comment by jokoon
17 hours ago
I don't understand how an LLM is able to reason about those things
LLM use language, but it can't "think" about biochemistry
I saw that LLM have reasoning capabilities, which is different from machine learning, but I don't understand how it works.
An interesting talk I heard at a conference once, that I can neither remember the speaker for or speak to their legitimacy, suggested that we might have some lower form of intelligence encoded into our language. They posed the idea that we have enough unique words, and combination of words, that it starts to have reason unto itself similar to how our neurons and their connection breed intelligence. The idea was that we as humans have baked intelligence into our own speech patterns. It seemed a little to abstract for me, but potentially goes a little way to explaining how a statistical averaging algorithm with some randomness, at scale, starts to look like it very occasionally has a genuinely novel thought.
In The Ticket That Exploded, William S. Burroughs proposes language is a virus in itself, coming from the Outside, and infecting the host with it's control logic. In Radio Free Abemuth, Philip K. Dick attributes a similar possession to a benevolent force, akin to the divine Logos flourishing intelligent development. Both seem open to an impersonal agency that maps to intelligent systems encoded in their transfer protocols.
The english language pattern is definitely shaping our thoughts and limiting our ideas. Just the whole idea of going from some abstract thought to actually making it out in tangible language, I mean no matter the thought it's a lossy transfer into a medium that lacks all the dimensionality of subconscious thought, neurotransmitter action, sensory information, and physiological response.
There's also something to consider with lower level vs higher level abstractions in language. E.g. jargon. One short word could have a 200 page thesis behind it defining all the ramifications. Talk about compression of information.
Now imagine if our language lacked say the mechanism of jargon, of using some meta word to define thousands of stringed together words at once. Every idea like "car" would have to be described from first principles. The species would probably never develop technology with this sort of language pattern present. If we could somehow level up beyond our current abstraction level, maybe that would make us even smarter, able to handle bigger ideas quicker in real time.
Even more simply than all this: I can only speak about what I have english words for.
3 replies →
That idea is also present in Snowcrash (and the Bible?) in a slightly different form - that our current fragmented languages are a way to protect us from the 'mind control' that results from having a single shared language - with some more layers of fantasy on top.
At one point there was a universal language, and later either recreated or scrambled, allowing for humans to have many languages for increased confusion.
Now that we are once again attempting to unify our language we find ourselves in a pursuit to build something to escape the Earth.
4 replies →
Language are the tokens in the inference output. How they are ordered comes from the weights, and the weights come from the microtubules holding and collapsing quantum state. Orch OR.
Sage thoughts, trash88
https://inv.nadeko.net/watch?v=Or_3tlEOLj4&pp=ugUEEgJlbg%3D%...
> some lower form of intelligence
Anecdotally, but I have lived in different cultures with entirely different languages and/or dialects, and the thoughts and even entire categories of thoughts people from these cultures express, or can easily express, are very much shaped by their language. Relatedly, I've also often witnessed multilingual people switch out of their native language to a second one just to express a particular idea or nuance, because they can do it with two words in that other language but would need at least a couple of sentences to say the same thing in their native one.
We use formal language to express symbolic relationships, e.g. "A implies B". But even "A implies B" has multiple meanings: material conditional, strict implication, logical entailment, etc. So, symbolic systems are not "pure and hard", they are also contaminated and softened by the vagaries of language outside them, which is our primary access to those systems: "valid" natural language and its strings of words. A statistical system that can string words into valid(=allowed by the distribution) language asymptotically approaches reason. So, the mind is not in the words, but in the laws that permit many words to come together, i.e. the probability distribution.
I thought that was how most people understood LLM’s capabilities? We have spent millenia creating language to map onto our world. Therefore, implicit in that language is a simulacrum of our world.
I think put more simply, you can say that humans wrote things down that were proxies for complex, physical phenomena in the real world. If you just look at what we wrote, you can recover world models that “understand” deeper patterns, bc the training data was only ever a proxy.
I feel like I can feel this happening in my mind in real time. Something like: the part of my brain that thinks thoughts is fairly rudimentary, basically just impressions or hunches--but then there's another part which translates them into words and grammar, and when it takes an impression it can translate it into something fairly sophisticated and intelligent, because it's somehow necessary in order to create a sentence which actually captures the impression.
An interesting theory that would cast some of the more interestingly phrased verses of the Bible in a new light, e.g.
“In the beginning was the Word, and the Word was with God, and the Word was God.” - John 1:1
"Word" there is Logos:
https://en.wikipedia.org/wiki/Logos
There's a lot more to it than just "language."
Does that help explain why learning a word for something can help understand the concept of it?
Does that mean the language(s) we speak determine how intelligent we are? Could learning French, for example—often considered a more expressive language—make a native English speaker more intelligent or even more compassionate?
It's arguable; see the Sapir-Worf hypothesis in the strong and weak forms.
I would argue the languages with the most colors belong to the most intelligent.
…or they just use a knowledge graph under the hood and don’t publish about it.
Isn't it more to do with being able to describe truth with language
Very intriguing, brings to mind Sapir-Whorf a little bit. You wouldn't happen to remember the name of the speaker or the conference, would you?
Does that explain why different countries that speak different languages have different engineering cultures? Like is german better suited towards engineering than english for example?
Michael Levin, the Biologist.
https://inv.nadeko.net/watch?v=Or_3tlEOLj4&pp=ugUEEgJlbg%3D%...
Interesting. Would this apply to any rich enough system of expression, like music or art? Or is there something specific about language that makes it different?
I know nothing of this topic but I'd say language had the benefit of much higher precision and flexibility of description than just music without words, though I guess if someone made a ai that used fragments of sound waves as the basis for its language instead of words you could argue it could be similar. But again it probably won't sound like music.
It's pretty clear reading from these comments that most HN members have a 2023-era impression of LLMs.
Modern chain-of-thought models with RL post training on verifiable tasks + realistic environments + rubrics are worlds apart from models trained on a simple next token prediction objective.
More money goes into the rubrics and RL environments than individual training runs themselves.
(Yes, at inference-time LLMs still output words one at a time, much like human speakers. But don't confuse the mechanism with the training objective.)
Even with heavy RL post training and rubrics, the model is still fundamentally bound by the next token prediction mechanism at inference. Rlhf and cot just affect the probability distribution of which tokens get predicted next. Take away the heavy agentic scaffolding and external feedback loops, and a single hallucinated token can still derail the entire chain of thought.
> and a single hallucinated token can still derail the entire chain of thought.
Incorrect. As the OP said, that is a very 2023 understanding of how LLMs work.
Grab a new model from OpenRouter. Have it work on a task. Change a few tokens and have it continue the completion.
6 replies →
When you speak or type, you speak one word at a time. When you move, you actuate one muscle at a time.
Does this mean that a single incorrect word or twitch will completely derail the task you’re trying to performance? Or will you, like any other intelligent being, recognize it and compensate?
1 reply →
> Take away the heavy agentic scaffolding and external feedback loops, and a single hallucinated token can still derail the entire chain of thought.
With reasoning models, a derailed chain of thought can be rerailed.
1 reply →
But but but....I was told it was a stochastic parrot! I liked that idea because it appealed to my vanity, and it described the gibberish produced by older models with bad prompting, and that was enough for me thank you.
/s
Nobody knows how it works, really. It just turned out that if you try to predict the next word then you get intelligent behavior, depending on amount of training data, and the size and topology of the network. But again, nobody knows why, and what the limits are.
Agreed. We went this direction for our golems, djinns, and other mechanistic minds because we believe it sort of reflects the primitives of our own neurons (which we also don't fully grok).
Linus Torvalds:
~"Predicting the next token is not an insult. It's pretty much what we all do."
That's of course absurd. The human brain doesn't represent information as discrete tokens, nor does it form sentences autoregressively.
2 replies →
I heard someone who studies this sort of thing say basically what biological neurons are trying to do is predict as well. Predicting what exactly? I’m not sure. The next time they should fire or something. I can’t find the YouTube video now.
In the last few decades, there has been an increased interest in the role of prediction in language comprehension. The idea that people predict (i.e., context-based pre-activation of upcoming linguistic input) was deemed controversial at first. However, present-day theories of language comprehension have embraced linguistic prediction as the main reason why language processing tends to be so effortless, accurate, and efficient.
https://www.earth.com/news/our-brains-are-constantly-working...
https://www.psycholinguistics.com/gerry_altmann/research/pap...
https://www.tandfonline.com/doi/pdf/10.1080/23273798.2020.18...
https://onlinelibrary.wiley.com/doi/10.1111/j.1551-6709.2009...
1 reply →
Predict activations in adjacent neurons, roughly; the relevant keyword is "Hebbian learning" (and "predictive coding" at a higher level).
1 reply →
Predicting reality, under the "controlled hallucination" framing, corrected by sensory error signals. The brain has no access to ground truth, only input data that helps correct the hallucination.
Based on lots of human interactions, I think there are a lot of human beings out there who mentally aren’t much more than “next word predictors” who happen to be made of meat+neurons instead of silicon+code.
That’s my and probably most people’s understanding.
I have a feeling we know more than that about how it works.
We don't know how our own intelligence works, much less anything else's
3 replies →
Language (human and computer alike) is excessively redundant. Read any sort of chain of logic or debate from somebody and you could sum it up, quite accurately in about 5 words. The rest is either fluff or supporting statements that should flow naturally and logically from the initial premise. My own post here is a perfect example. Everything I said after the first few words is little more than dumping directly connected statements.
Train on a massive body of text, figure out what correlates with what, and next thing you know you have a rather impressive facade of logic that can even connect things in novel ways where a connection is clearly called for, but not yet made. I call it a facade because LLMs will be able to advance knowledge significantly in finding these clear connections, but they exist only because no human can hold more than a tiny percent of all knowledge in their own mind.
Where I expect they will run into issues is in finding the unclear connections - like going from an existence where math doesn't exist, to one where somebody 'invented', or more aptly - discovered, math. That's inventing something from nothing, rather than just logically connecting pieces. I don't see how this is possible with a token prediction algorithm.
Anyhow, the point I'm making is that language itself includes encoded logic. And so LLMs working as token prediction algorithms are able to exploit this functionality to produce statements that offer a facsimile of logical reasoning under a constrained domain.
I am no expert, just curious:
What is it that makes something truly novel or creates something from nothing?
When we do it, do we apply existing concepts, combine them with a general intuition for how physics work in the real world, and use that to form a hypothesis that we then test in experiments?
Again I think the example of math is good. Many isolated tribes still don't even have numbers. They simply refer to things in broad quantifiers like - none, one, few, some, many. And that's perfectly fine for their needs! Many of the problems that you need math to solve - or that lead naturally to math, like currency, only exist once you've already discovered mathematics.
So try putting yourself in this ancient mindset before mathematics. How did somebody invent it, come up with the concept of numbering everything, further develop the various 'tricks' for manipulating these numbers, and so on? In terms of raw 'complexity' it's far less impressive than the latest LLM models solving some obscure mathematics problem that almost nobody understands.
But in terms 'intelligence', I find it vastly more impressive - because it's again this sort of difficult to describe concept of going from nothing to something. There is no logical baseline that naturally and cleanly leads to math. Almost like a child would say when asked how they learned something, 'Oh I just thought it up.' Except in this case, somebody genuinely did!
2 replies →
The cure for HER2- metastatic breast cancer is a simple matter of ...
Please predict the next word.
Intelligence is implicit in language understanding. The best possible next-word-predictor is omniscient.
This whole thing is an example of why the philosophy of this stuff is so fun. The trick here is buried in the word "is".
Just for kicks, I actually put your sentence into an LLM. The response was along the lines of, "Your query was incomplete and about medical knowledge, so I need to be careful. There is currently no cure..." and then goes on to do a decent job of summarizing existing treatment approaches for metastatic breast cancer.
What's so interesting about this is your notion of prediction here is divining the answer in reality, i.e. finding a cure for breast cancer. But its notion of prediction is determining the next logical sequence of words given its training set, so it produced a block of useful and context-relevant text, but not what you actually care about. This leads into the much broader question of what do we mean by "intelligence," which forms do these things have and not have, etc. etc. If nothing else it's all very fun to think about and debate.
> The best possible next-word-predictor is omniscient.
Omniscient for the set of "meaning" embedded into it's training set. It's not broadly omniscient, big difference.
Sure, but same for any intelligence. The "training set" is just "the environment of adaptation".
2 replies →
Right, but what's the limit of what you can deduce computationally from truly vast training sets? How much structure is there in the subtext of what's written down? It looks like there's rather a lot.
The cure for HER2- metastatic breast cancer is a simple matter of [intensive well-funded research]
That wasn't too hard, maybe I'm superintelligent?
What does omniscience have to do with reasoning? If you know everything, you don’t have to reason. But these next-word-predictors aren’t omniscient.
Omniscience doesn’t imply that you have to store all the information, but that you can retrieve/reconstruct it, and reasoning allows it. In fact would be impossible for any physical intelligence to store all the information as plain as it is infinite.
Here's my grok of it: Deep learning models progressively abstract a concept presented at the input by passing the input through many sequential layers () until an output layer transforms the output of the final layer into something interpretable, such as an indication of what token to predict next, or a classification, or whatever. The transformer architecture futhermore offers layers that allow different parts of the previous layer's output to sort of mix with each other in complex ways. As you get into greater levels of abstraction, the attention process is mixing very abstract concepts with each other in a nonetheless highly structured manner. I believe this is where the intelligence lives.
sometimes with residual connections, but we can ignore that for sake of simplicity.
Intelligence as a measure of the ability to define predictive models of certain problems (and their solutions).
Promoting LLMs is encoding the problem we want into the query vectors, and through the magic of the complex training and the power of operations in a very large dimensional abstract space the AI can manipulate the representations, and iteratively approximate solutions. (And using bigger and bigger contexts and better encodings it can form better models.)
Language emanates from intelligence. That means the patterns and structure that make up human intelligence will appear in language. LLMs are created through so much language training that they can approximate (and now to some degree exceed) human intelligence using pattern recognition, statistics, and autocomplete (in layman’s terms).
Not sure how it is now, but early “reasoning” was simply the big labs sticking “wait a minute, what if I…” type language blocks into the process to trigger something like our own internal reasoning.
I'm not an expert, but my current mental model for this sort of thing is that the thoughts were already there, somewhere in the training data.
Some human was looking for something like this once. They didn't find it, but they wrote about the search precisely enough that the finding can happen during inferrence.
Maybe somebody will come along and school me, but for now it's a fun way to think about it: A million dead ends, each with a uniquely disappointed human, now with a chance at a second life in the hands of a different human they haven't met. If only the weights had encoded enough to introduce us, supposing they still live.
Compression and understanding are correlated.
How do you think? I think with words.
Do you have an internal monologue?
I don't. I seem to think at a more abstract, pre-verbal level rather than through an internal voice.
Some studies suggest that frequent internal monologue may occur in roughly 30–50% of people [1], but the research is based on relatively small samples.
[1] https://www.psychologytoday.com/us/blog/intersections/202304...
FWIW I have ADHD and my internal monologue just will not shut up. I don't exclusively think in words but I do think in a lot of words.
I honestly don't think that people's self reports of whether they have an internal monologue are super reliable. They can mean different things by it or be psychologically attached to a particular representation of their thought processes, it can be related to self concept or self esteem or personal perspective in ways that are hard to tease out.
I think something like this proved to be true when it came to folk theories of different learning styles (e.g. visual vs language based) once those started being tested in rigouros ways. People could still be right but I would be interested to see what we get if we test more directly for subvocalization or fmris for language based brain activity.
I have different modes depending on what I'm doing, but I think usually I'm reasoning non-verbally
If you try and think before you speak, does it just not work?
1 reply →
I tend to agree with Albert Einstein below; there's a very physical/spatial aspect to my problem solving before it can be translated to words. I work in software so there's nothing innately physical about it. Never put much thought to it until LLMs brought it up for debate.
"The words of the language, as they are written or spoken, do not seem to play any role in my mechanism of thought. The psychical entities which seem to serve as elements in thought are certain signs and more or less clear images which can be "voluntarily" reproduced and combined....From a psychological viewpoint this combinatory play seems to be the essential feature in productive thought....The...elements are, in my case, of visual and some of muscular type. Conventional words or other signs have to be sought for laboriously only in a secondary stage, when the mentioned associative play is sufficiently established and can be reproduced at will."
Theres more than words in our minds. Think harder are you absolutely sure? You REASON with words but your ideas dont form just from you reasoning. The ideas just seem to come out of nowhere to the part of your brain that then reasons around them.
same for llm though
Fair, but the original comment was about reasoning.
How do you know that it's the words driving the thinking, rather than the stream of words just being an observable trace tacked onto the actual thinking?
These days, words. When I was in an environment where language swapping between 4 to 5 languages was common, I thought in pictures and described it in the correct language for the audience. It was a plasticity mind trip.
Also saved pesos on the charge-per-text SMS schemes the local phone companies used because we could embed information across so many options.
You think with and without words. When you have to pee, it isn't like you speak to yourself "Gee, pinch in the loins, I guess that must mean must have to pee. Alright legs, get me up off my butt. Left right left right left right. Stop. Hand, get the zipper going. Johnson, your turn now."
Nope. You up and pee.
My friend has aphantasia and cannot think with words, sounds, or pictures.
How do they describe how they think?
3 replies →
Biochemistry is abstract to us humans too, we can only create hypotheses, and validate them experimentally.
And for those hypotheses - we use language.
Most human reasoning happens within language - even mathematics is an abstraction that allows us to map concepts we don’t natively hold into a linguistic processing layer.
AI is way beyond conventional LLM architecture now. It combines LLMs with search + RL. The traditional LLM architecture hit a wall around GPT-4o. Arc AGI evals show this.
All that extra is clear as day compared to the mystery of how neural network training decides to divide and balance the weights in even small neutral networks.
We can, at best, approach a good set of weights, even in tiny neural networks.
Imagine if we found a way to calculate the exact optimal weights for a given loss function. I mean, there is an exact optimal solution, it exists, but we can't find it exactly, even for a neural network with just 50 parameters.
There is no point in that because the loss function itself is already an approximation. No one knows what is the exact loss function for any given non-trivial real-world task.
2 replies →
I don't think LLMs currently have direct reasoning abilities, but as we make them more complicated (MoE, RL) I think we're getting better at learning an implicit world model that guides the token output distribution towards making good hypotheses.
If LLMs can recursively improve and redesign themselves, it may be very difficult to tell when they have quietly crossed the technological singularity while concealing their true capabilities and intentions.
LLM's are giant cross-domain search engines. Not thinking machines. They can discover patterns extremely well. This discovery is well within that space.
No worries, no one does. Exactly like no one knows how the brain reasons either.
I imagine it's writing a story about a character doing those things and then reading the story and acting on it.
Granted, I feel like munging gigabytes of text data (i.e. G, A, T and Cs) would be something LLMs would be good at
Ha ha, you completely understand how you work though.
Stephen Wolfram had a great description of this effect in the early days (GPT 3.5 era):
Machine learning trains the network to do... anything that you reward it for. If you keep training, it keeps getting better.
Next word prediction can always keep getting better.
At first, simply "learning" spelling is what makes the predictions better because tokens are word chunks, not always whole words.
Then, the models "run out of steam" and can't get any better by learning more spelling rules, but the gradient descent forces them to get better... so they do... by learning the rules of grammar.
At this point the AIs can output correctly spelled and grammatically coherent sentences, but the sentences ramble on about nonsense topics.
So what happens next as the models run out of grammar rules is that they're forced to learn the rules "above grammar": logic, world knowledge, coherent story telling, etc.
At some point they learn to output pages and pages of fluid, coherent text, but... if they're not smart, if they don't think, and if they don't know what they're talking about, then they're still "suboptimal" and their forced gradient descent will make them close those gaps.
Eventually, the only way they can improve at "next token prediction" is by building up to human-like intelligence, including an inner monologue, theory of mind, and everything.
We can even read their "thoughts": https://transformer-circuits.pub/2026/workspace/index.html
What's to say it "can't" think? These are not tasks you can do without thinking.
I don't see why its that crazy that a system with a huge amount of parameters starts to exhibit emergent behavior
I don't think anyone knows, not even the LLMs.
I mean, the subtlety of the neural network weights that emerge from training are not fully comprehended by anyone, man or machine.
Every individual calculation is understood, and every step of training is understood, but the exact nature of those weights that divide the responsibility of responding to subtle changes of input in intelligent ways is beyond me.
It's really good at pattern recognition.
So I'm not sure how it knows to be 'surprised' that alone is pretty fascinating.
If I were to guess, being pleasantly surprised is just a learned appropriate social response from the expectation of receiving a reward and as such, that social norm is codified sufficiently enough in our writings that it appears in LLMs output.
It’s sort of like all the people who will ask Claude or GPT to validate their complete nonsense and receive unyielding praise for it, the models just learned that this is the best received response based on training data and RL.
I bet these same sorts of expressions can be found in practically every failed attempt as well.
[dead]