Comment by jsemrau

You are on the right track in my opinion. The key is to encode the interface between the game and the agent so that the agent can make a straightforward choice. For example, by giving the agent the state of a nxn board as the world model, and then a finite set of choices, an agent is capable of playing the game robustly and explaining the decision to the game master. This gives the illusion that the agent reasons. I guess my point is that it's an encoding problem of the world model to break it down into a simple choice.

[1] https://jdsemrau.substack.com/p/evaluating-consciousness-and...