Comment by hackinthebochs
2 hours ago
So you weren't giving it an updated board state after every move? If you want to compare apples to apples, it should give an updated board state for each move, or you should play blindfolded.
2 hours ago
So you weren't giving it an updated board state after every move? If you want to compare apples to apples, it should give an updated board state for each move, or you should play blindfolded.
I can play blindfolded. I am expert OTB (though I haven't played in a while). The game was like 18 moves of theory in the Maroczy Bind.
An LLM has been trained to do everything it does blindfolded, "only" using perfect recall of everything in it's hundreds of thousands of steps of context, and hundreds of layers of KV cache. It's a computer - it has a massive advantage over a human.
The fairest apples-to-apples comparison of an LLM whose training data included chess games would be a trained human such as Magnus Carlson, who can quite happily play a dozen or more simultaneous blindfold chess games.
Blindfolded flex by OP aside (I can barely play when seeing the board), considering reasoning traces and their nature, if we want to be fair, a person would have to get the moves, but be allowed to write them down or draw up a board in their notepad. My working memory can barely handle five chunks, a models reasoning tokens are masses of written text in comparison.