Comment by YeGoblynQueenne
21 hours ago
Eh, no, that's not right. I might need to brush up on my Sutton & Barto but the RL task is traditionally defined as, informally, "given a current state observation predict the next action, state and reward". A policy is always predicting the next timestep's reward. Otherwise, how would it know what to do next?
Brush up :)
The policy is optimized to maximize the the total reward, defined as the sum of the reward at each step, discounted by some factor.
Alright, I'll have to check up on that. Thanks for being nice about it.
Hey man, I'm not crazy. Yes, the goal is to maximize the cumulative reward like you say but to do that a policy (agent) has to take the actions that maximize its expected return in each time step. That's what the discount factor applies to, the expected return.
To maximize its expected return it has to predict that taking action α in state s in time step t will produce state s' and reward r in time step t+1. Which btw it can predict by estimating a value function that sums over the expected return from time step t onwards. The value function essentially tells the policy what state it wants to be in and what action to take to get to that state, in order to eventually achieve its goal (represented by max cumulative reward at the end of an episode).
So, yes, like you say, the whole policy is a game result predictor but the decision making process is a next-move predictor. The policy must make the locally optimal decisions to get to the globally optimal result, if I may be so bold. It's the same problem with a different solution as in heuristic search, except you have a reward function instead of a heuristic cost function.
Sutton & Barto 2ed is free here btw:
https://web.stanford.edu/class/psych209/Readings/SuttonBarto...
The value function stuff is in Section 3.7. Now if someone could explain the grid world example in Figure 3.5 to me that would be great because I'm very confused about the fact that any action at state A or B takes the agent to state A' or B' respectively, since those are not even adjacent to A and B. I'll go ask chat I suppose.