Comment by thom
2 hours ago
I've not noticed this happening if you give it the FEN each move. The alternative is just blindfold chess and very few humans can do that for long.
2 hours ago
I've not noticed this happening if you give it the FEN each move. The alternative is just blindfold chess and very few humans can do that for long.
I haven't tried it myself, but people seem to report that the illegal moves surface eventually. It just takes longer: https://news.ycombinator.com/item?id=49720751
Nothing is forcing the LLM to play 'blind'. If it's smart, it should be able to create its own representation of the chess board and update it with every move, just like a human would. Any chess engine that's sensitive to how the moves are formatted is clearly not very capable.
A human wouldn't do that, they'd look at the board. I'm not disagreeing that to demonstrate clear superhuman ability the LLM should be able to do this, but it plays better than most humans blindfolded, and with fair prompts seems very good otherwise.