Comment by little_endorian

You can take LLMs out of opening knowledge by playing chess960, and their performance degrades significantly. I just tried playing Claude Sonnet 5 (high), and it made its first illegal move on move 5.

They played 4...c6, followed by 5...Nc6, somehow forgetting about the pawn the just put on c6. (My move in between was 5. Nc3, and apparently they were trying to mirror me.)