Comment by danielmarkbruce

The policy is not learned token by token during RLHF and RLVR. The reward model doesn't score token by token.