Comment by diehunde
12 hours ago
AI bros: the LLM beats humans at solving Navier-Stokes and some old cypher. We are close to AGI
Also AI bros: LLM can’t beat an avg chess player. But that doesn’t mean anything. It doesn’t count
12 hours ago
AI bros: the LLM beats humans at solving Navier-Stokes and some old cypher. We are close to AGI
Also AI bros: LLM can’t beat an avg chess player. But that doesn’t mean anything. It doesn’t count
>LLM can’t beat an avg chess player.
Why should that matter?
It depends on what you are selling it as.
It only matters if you are claiming it to be general purpose.
If you admit that it's just a collection of narrow capabilities - whose strength is mostly confined to the 1000 or so RL environments it was post-trained in, then there is of course no expectation of it being general purpose.
The AI companies seem to heavily want you to believe it is some some near human level general intelligence, so therefore pointing out all the things it can't do is very relevant.
If something has general intelligence it should be able to read the rules of a game and follow them. Therefore an artificial general intelligence (AGI) should be able to do this.
So we have a situation where very powerful and influential people are saying we will have AGI in 6 months (if we don’t already), yet the facts on the ground are so clearly pointing in the opposite direction.
I would bet a lot of money that Astra can follow the rules of chess (perhaps if repeated within the context window). Also, this is a different argument than what I responded to.
4 replies →
So we humans are not a general intelligence then?
And the stuff i'm using LLMs daily is just fake?
I see i see. I will see myself out of this weird discussion while I let an LLM continue doing a lot of interesting things.
3 replies →
> Why should that matter?
Because we want to use this as a replacement for humans, and the average human can learn the rules of chess without needing to see the rules explained hundreds of thousands of times in millions of games.
So, yeah, it matters if a model has millions of examples of something in its training set and still cannot follow the rules.
We're not talking about learning the rules of chess here, but playing a competent game from just being shown the rules. Why is it so hard for people to keep track of the thread of discussion?
3 replies →
The fact that LLMs can play chess at any level is a strong indication we are in AGI.
It would be more impressive if they could play chess (or do anything they haven't been custom RLVR trained for) by reasoning, rather than just "have a go at it" prediction which is closer to memorization.
HOW you do it makes a big difference in how you should assess the capability of the thing doing it. Stockfish will trounce any LLM, and any human, at chess, so should we say that Stockfish is smarter than both?
This is roughly comparable to observing a cat batting a ball away with its paw and taking this as a "strong indication" that cats can play any sport.
Yes, a good analogy. Except the cat actually follows the football rules and can beat some humans. And has no physical limitations to play other kinds of sport that you might imply.
Can they if they frequently make illegal moves?
Do they?
No it isn't. Computers could play chess long before LLMs, better than LLMs can in fact. That didn't make them AGI.
You are saying "No it is not" without an argument. The fact that computer systems could play chess yet not being AGI has no relevance to LLMs' ability to play chess being AGI, because the point is about G, not I. There's little doubt about A or I parts.