Comment by bigstrat2003
14 hours ago
> Frontier labs don't care about chess. If OpenAI cared, GPT-7 could be a grandmaster+ level chess player.
If the models were actually intelligent, the way that the boosters claim, they wouldn't need to be tuned to play chess in order to be good at it. That's kind of the point of intelligence, that it is generically applicable to whichever task one wishes.
Thats just absolutly not true.
A human being has general intelligence and needs A LOT of training and finetuning to become good in chess.
And there is a relevant and significant difference between the expectation of an AGI and an ASI system.
Humans don't need a lot of training and finite tuning to make only legal moves.
An intelligent adult could simply read a short summary of the rules of chess and then, if they were careful, play a very bad game of chess without making illegal moves.
An LLM that has not been trained on any chess data cannot do that, at present. If you doubt it, take a current model and tell it that you want to play it at a variant of chess where, say, knights can also move diagonally like bishops. A human can easily adapt to this new ruleset (even if they make tactical mistakes, not having practiced with this variant of the rules).
How long a prompt do you think would be required to cajole an LLM into making legal moves at the rate of a human? Or do you think no amount of prompting could do that?
6 replies →
Pretty much this. Feed it a book or two on chess, and you should have a decent (or good) player. That's the generic intelligence people have. The aims is not to be supremely talented at something, but being able to read a manual and figure how to use/play something. Mastery can be gained overtime.
1. The LLMs have surely ingested hundreds if not thousands of books on chess.
2. The study (along with other posters here) show the models can’t even stick to following the rules of the game
If you gave a human a book or two on chess they would not become a decent player (they would be closer to 500-600 than 1100 ELO) and they would only get better after playing hundreds or thousands of games (often making illegal moves and moves that violate the rules of chess as they learn).
Your assumptions/intuition about generic human intelligence feels quite incorrect, considering LLMs currently play better than a brand new human player would (presumably without any attempt to fine tune them specific on chess, such as playing thousands of games).
the discussion isn’t really about whether language models can become strong chess players though, the point is they seem to struggle to consistently make valid moves. Most humans don’t need to read two books to pick that up, just a couple lines of basic instructions
2 replies →
That's quite untrue. I taught my (adult) brother the moves, the only illegal move he ever tried against me (over his 6 first games) was a castle with a rook that already moved twice. Within a few hundred games (less than 500 for sure, he played 3 minutes blitz but always took at least 10 minutes analyzing his games) he was rated 1100 on lichess (which is like 1050 on chess.com and unranked in the real world).
2 replies →
> considering LLMs currently play better than a brand new human player would
They’ve ingested all the literature on playing chess, a brand new human player has not.
4 replies →
If humans were actually intelligent, they wouldn't need to train and practice to play good chess. I mean, what level do you think people without any practice or training are ?
Except all these LLMs were already trained with hundreds of chess book and game databases and they still suck
If all you do is read chess books, you'll be a shit player. Training and practice is what it takes to be great.
6 replies →
Contrary to popular belief, you need a lot of training on something for an LLM to be good and consistent with it.
People think that if one mention exists in the training set, then the LLM is perfect at it.
1 reply →
OpenAI making the next model good at chess is not analogous to a human training to get good at chess. It is analogous to God creating Human 2.0 which now has increased chess playing ability. If LLMs were intelligent the way humans are, then the models that exist right now would be able to spend time improving themselves at chess and become good at it. They can't do this because they are not, in fact, intelligent.
[dead]