Comment by TheOtherHobbes

I'm not sure why anyone is expecting stochastic systems to be deterministic.

Chess is a deterministic game won by a combination of known movesets and constrained multi-level forward search.

LLMs do neither of these things. They don't reproduce training data exactly, their next response is more 'inspired by' prompts and its own memory than produced deterministically, and they don't have the capability to do general forward search on their own.

So when you ask an LLM to play chess you're getting the equivalent of a very compressed and lossy JPEG of chess rules and strategies with added per-turn random noise.

They also don't have the ability to design their own chess engine, although it would be interesting to see what happens if you ask for one.