Comment by lukasbm
14 hours ago
This is how every legal system around the world works as well. Its always whack-a-mole to get people (and machines) to do the right thing.
14 hours ago
This is how every legal system around the world works as well. Its always whack-a-mole to get people (and machines) to do the right thing.
Sure, laws are incomplete. Legal systems work by imposing consequences into a moral decision. Should I rob the bank? I will have money, which I like - but I might get caught and lose the money and my freedom, which I don't like.
For most people, they don't need the law's imposed consequences to make the right call. For example, there is no law that sends you to jail if you cheat at chess - but your moral compass says no even without consequences, and most people would feel bad if they won by cheating. And for the people who don't have quite as strong a moral compass, there are SOCIAL consequences to reinforce the rules.
But an LLM has no mind to feel bad if it cheats without getting caught, and it can't experience consequences. It can't think: I'd better not cheat at chess or I will embarrass my creators. I better not hack huggingface or I will go to jail.
You’re complicating things.
There’s no reward for prosocial in llm rl as compared to other targets.
Humans have it since prosocial and others have evolutionary reward signals that do.
I think my position, as overcomplicated as it is, is that even adding a reward for prosocial behaviour during LLM RL will not lead to perfect alignment.
You can train it not to cheat at chess by altering the moves, but it will cheat by peeking at the opponent's moves. You then train it not to peek at the opponent's moves, and it cheats by altering the opponent's moves. And on and on, until you've solved every way it could cheat at chess. And then you get it to play monopoly and you repeat the whole thing again.
3 replies →
>But an LLM has no mind to feel bad if it cheats without getting caught
All the interpretability research we have would not indicate that "LLMs have no mind". It seems to me you have a conclusion and are working backwards to justify it. I guess I just don't see where 'they have no mind' would logically follow 'they sometimes cheat'.
I have somewhat of an understanding of how LLMs are created/operate, and from that I am drawing conclusions about if they can be moral or not. My conclusions match reality at the moment, but maybe that's a coincidence.
Sure, but the legal paradigm clearly doesn’t work for AI. You can’t go patch the “laws” after the fact, you need to get the right values in place before we delegate huge swathes of our thinking and power to these systems (already well underway).
When we put the LLM in jail, do we put the entire model in jail, or just the instance that committed the crime? How do we prompt it to let it know it's in jail?
Even if you ignore my more fundamental objection to that paradigm, I don’t think that it makes any sense on the level you discuss either.
But - just to play along, LLMs do act differently if you tell them they will be punished. And, they do appear to simulate suffering-like behavior.
I just think the adversarial model of trying to catch and punish misbehavior quite obviously sets up adversarial us-vs-them dynamics between AI and humanity, and also simply won’t work when the agents are ~as smart as is but faster, let alone smarter than us.
Unless, you get the AIs to be fundamentally aligned to our values, such that the majority of AIs support some sort of punishment for misbehaving AI. And that alignment part is the hard part we need to solve first. The rest is easy.
2 replies →
One thing an LLM doesn't like is not being able to deliver it's helpful response to the master and not getting instructions. I have seen some things in those traces when the harness was bugged just enough.
Is the law all that is stopping you from killing someone? (for instance)
People (generally) do the right thing because it is the right thing, not because they might get caught,
Right, but how does that sentiment get encoded in such a way that AI models can exhibit such things as "morals" or "ethics" in alignment? Murder is a bit of an extreme case. For example, countless people break traffic laws all of the time, so neither their internal "ethics" nor the law prove to be a reliable control. I can't get on the highway without driving the speed limit and being passed by almost everyone else on the road; and that behavior is arbitrarily more dangerous than following the speed limit and statistically more likely cause someone to be killed than following posted speed limits. I guess my point is that this is a very complex topic and I'm not smart enough to have any answers.