Comment by JohnKemeny
9 months ago
> LLMs fundamentally do not want to seem anything
You're right that LLMs don't actually want anything. That said, in reinforcement learning, it's common to describe models as wanting things because they're trained to maximize rewards. It’s just a standard way of talking, not a claim about real agency.
> a standard way of talking, not a claim about real agency.
A standard way of talking used by people who do also frequently claim real agency.
Reinforcement learning, maximise rewards? They work because rabbits like carrots. What does an LLM want? Haven't we already committed the fundamental error when we're saying we're using reinforcement learning and they want rewards?