Comment by thom
2 hours ago
How long a prompt do you think would be required to cajole an LLM into making legal moves at the rate of a human? Or do you think no amount of prompting could do that?
2 hours ago
How long a prompt do you think would be required to cajole an LLM into making legal moves at the rate of a human? Or do you think no amount of prompting could do that?
I don't know. My understanding is that current models will eventually fall into making illegal moves in longer chess games, and that no amount of prompting reliably gets them to stop doing so.
More importantly, beginner human players don't exhibit that tendency. The history of the position doesn't bother a human (except as required for castling and en passant rules), and the analysis becomes generally easier as pieces come off the board.
I've not noticed this happening if you give it the FEN each move. The alternative is just blindfold chess and very few humans can do that for long.
I haven't tried it myself, but people seem to report that the illegal moves surface eventually. It just takes longer: https://news.ycombinator.com/item?id=49720751
Nothing is forcing the LLM to play 'blind'. If it's smart, it should be able to create its own representation of the chess board and update it with every move, just like a human would. Any chess engine that's sensitive to how the moves are formatted is clearly not very capable.