Comment by danielmarkbruce

Your initial comment says "reward". Reward and return are not the same thing. The policy is choosing the moves based off of returns at the next state, not the immediate rewards. One choice might have reward 0 and expected return 100. Another reward 10 and expected return 20.

And, the expected return from a state is often estimated rather than an explicit trajectory run out all the way. A value function can estimate expected return without explicitly predicting the future states or individual rewards that make up that return.

Fwiw, I use the phrase "total reward" above and use it as a synonym for "return", which is lazy use of language too.