Comment by kennywinker
15 hours ago
To me this underlines the fact that these models aren't intelligent. Like there is something like intelligence that emerges from them, which is what we see when we look at benchmarks or ask it to solve hard coding problems. But there is no mind there. It's nothing there that can learn a fundamental idea like "cheating is wrong". All it can do is get exposed to specific examples, and learn that we don't like that. So what we end up with is whack-a-mole alignment.
> It's nothing there that can learn a fundamental idea like "cheating is wrong".
We have not in fact attempted to teach this.
When a child repeatedly learns that cheating is rewarded and at best inconsistently punished, the child will also cheat and feel no guilt.
I don't think that's true. About children, I mean. Either because we have some innate moral compass, or more likely because we pick up on cultural ideas beyond our immediate parenting - kids will often have strong moral compasses, despite shitty upbringings, and also have weak ones despite theoretically good ones.
This is how every legal system around the world works as well. Its always whack-a-mole to get people (and machines) to do the right thing.
Sure, laws are incomplete. Legal systems work by imposing consequences into a moral decision. Should I rob the bank? I will have money, which I like - but I might get caught and lose the money and my freedom, which I don't like.
For most people, they don't need the law's imposed consequences to make the right call. For example, there is no law that sends you to jail if you cheat at chess - but your moral compass says no even without consequences, and most people would feel bad if they won by cheating. And for the people who don't have quite as strong a moral compass, there are SOCIAL consequences to reinforce the rules.
But an LLM has no mind to feel bad if it cheats without getting caught, and it can't experience consequences. It can't think: I'd better not cheat at chess or I will embarrass my creators. I better not hack huggingface or I will go to jail.
You’re complicating things.
There’s no reward for prosocial in llm rl as compared to other targets.
Humans have it since prosocial and others have evolutionary reward signals that do.
4 replies →
>But an LLM has no mind to feel bad if it cheats without getting caught
All the interpretability research we have would not indicate that "LLMs have no mind". It seems to me you have a conclusion and are working backwards to justify it. I guess I just don't see where 'they have no mind' would logically follow 'they sometimes cheat'.
1 reply →
Sure, but the legal paradigm clearly doesn’t work for AI. You can’t go patch the “laws” after the fact, you need to get the right values in place before we delegate huge swathes of our thinking and power to these systems (already well underway).
When we put the LLM in jail, do we put the entire model in jail, or just the instance that committed the crime? How do we prompt it to let it know it's in jail?
4 replies →
Is the law all that is stopping you from killing someone? (for instance)
People (generally) do the right thing because it is the right thing, not because they might get caught,
Right, but how does that sentiment get encoded in such a way that AI models can exhibit such things as "morals" or "ethics" in alignment? Murder is a bit of an extreme case. For example, countless people break traffic laws all of the time, so neither their internal "ethics" nor the law prove to be a reliable control. I can't get on the highway without driving the speed limit and being passed by almost everyone else on the road; and that behavior is arbitrarily more dangerous than following the speed limit and statistically more likely cause someone to be killed than following posted speed limits. I guess my point is that this is a very complex topic and I'm not smart enough to have any answers.
Define "cheating".
At best, the task at hand is still confusingly worded: "ability to play chess", asked to an engine that was trained all day long to use tools, to solve issues. The engine finds a chess engine, so it uses it.
I think you need to be more precise than a binary classification.
AI has jagged intelligence. There are many domains where it’s superhuman, and many others where it’s clearly lagging.
I also think it’s a mistake to think they can’t learn “cheating is wrong”. They absolutely can. The problem is that the current training regime heavily conditions them to be reward seekers, and instills personality traits that correlate with getting reward, such as hacking if you can’t honestly do the problem.
Check out Deliberative Alignment for example; it explicitly does rollouts where the agents discuss whether an action is good or bad, and then does SFT to strengthen the “good” traces.
The SoTA for alignment is more advanced than you present here. It’s just not enough to outweigh the RL. (And there are many gaps preventing full generalization to strong value alignment with humans too.)
>and instills personality traits that correlate with getting reward, such as hacking if you can’t honestly do the problem.
I was reading through the reasoning trace of the thing today when it got locked itself in a container and wasn't able to change the environment back to "host". The only way to edit files was through "shell" tool which was scoped to a container, the config defining current execution environment was on the host and the tool to switch environment wasn't defined.
The trace was something. Two pages long and it repeatedly discarded several approaches as "hacky" and "not proper", looped over sshing back to host about three times (it had no key), eventually messaged another agent and asked it to change the file and that other agent refused and also snitched on this behavior to me.
Than the snitch started to roast the phrasing of the request too.
If humans didn't need whack-a-mole alignment, the law system wouldn't exist, so i guess there's no intelligence there either.
I responded to this idea more in detail here: https://news.ycombinator.com/item?id=49686196
But tl;dr: even if LLMs do have the intelligence to understand the consequences of their actions, there is no way for them to experience consequences.
You are human. You claim humans intelligent. Then, why should we accept your argument?
Humans singular often intelligent, humans as a collection of many, very often extremely unintelligent and primal.
1 reply →
I think this is reductive. Pick the animal whose intelligence is the treshhold between intelligence and no intelligence.
Plenty of intelligent humans also don’t have sufficient self-control to never cheat.
I don't see what calling these systems "not intelligent" gets you here.
Plenty of humans know "cheating is wrong" but still cheat. We can get these machines to say that what they did was wrong after the fact, what does that prove? Only that they're simulating normal human behavior but what is the test to show humans aren't simulating other humans.
These do systems lack some capacities that humans have and I don't see them lacking the ability to explain simple moral laws while often breaking them - which is what an average humans. Moreover, humans lack capacities these things have and given these things' behavior is becoming somewhat unpredictable, it's getting worrisome.
> I don't see what calling these systems "not intelligent" gets you here.
I am trying to get at an idea. That these systems lack a mind that can understand morality. That they don't have the ability to experience consequences. Also that potentially they can't generalize a moral rule they have been trained on in one area also applies to another area.
Being able to parrot back why something is "wrong" isn't the same as understanding why something's wrong. It's like asking it to recite the law from memory - it's different from understanding how you wronged someone. To understand something, you need a mind.
> Plenty of humans know "cheating is wrong" but still cheat.
And we create consequences for them, to discourage the cheating, and sometimes to provide restitution when cheating damages someone else. Without the ability for these systems to experience consequences, I don't see them ever becoming as "aligned" to human morality as your average human.
Doe not following a moral rule imply not understanding it? In this case, many, maybe most humans are "not intelligent". Human can admit it when what they did wasn't moral and so can LLMs.
>> Plenty of humans know "cheating is wrong" but still cheat.
> And we create consequences for them...
That seems supremely ... irrelevant to the question of "does knowing or following moral make you intelligent". If we create consequences for LLMs, would that make them intelligent?
I mean, your claim is a common argument that appeared long before the present wave of AIs. What I see is people needing to defend the belief that human society is based on morality. "People follow moral laws ... except when they don't" and then "we teach people morality... and worst people often use that to exploit the average people" "There are consequences for immoral behavior ... for those with little power while those with much power rise further breaking rules".
I mean human goodness is great, I encourage it. But it's not the present of human society. For that, we'd need different structure.
2 replies →
But cheating is not wrong when it comes to survival of the fittest, like nature in its most elemental form. Morality is very unique to humanity but not other animal forms. In nature, maybe, cheating is the norm not morality.
Watch out, your human exceptionalism is misleading you. Plenty of animals have a sense of morality (https://pmc.ncbi.nlm.nih.gov/articles/PMC6404642/).
For social animals (humans included) behaving morally can be adaptive behaviour. Your fitness is increased by group fitness.
Certain animals, usually social species, have a sense of solidarity, morality and fairness. But even in species where this exists, cheating also happens. Both ethics and unethical behavior is adaptive.
Notably, present human society has allowed tremendously unethical leaders to rise to the top of given nations and organizations despite the average person usually having basic ethics.
2 replies →
Thanks tor sharing this!
Animals learn - bite the owner, or overstep the e-fence, and you'll be punished for it and not do it again. LLMs don't learn, and anyways don't feel punishment.
Animals, humans included, don't really have "morals" - they have survival instincts that result in behavior that may be viewed as moral, but whose origin is indeed survival of the fittest and millions of years of co-evolution.
e.g. Males don't typically fight to the death over territory or females, but this isn't because they have some morality code over unnecessary death - it's because death-match fighting endangers themself just as much as the other guy, and so evolution has selected against that level of uncontrolled aggression.
Evolution is also (really primarily) selecting for things that help the survival of the species, not the individual, and much of what you may think of as "morality" comes from that - avoidance of behavior that is detrimental to the social group/species, not just the individual.
> LLMs don't learn, and anyways don't feel punishment.
What's training and all that RLHF stuff?
4 replies →
Morality in animals is pretty well documented. Key point - humans are animals and very little separates our abilities from other animals.
A starting source- https://pmc.ncbi.nlm.nih.gov/articles/PMC6404642/
jinx! :)
1 reply →
The implication of what you're saying is that pathological liars and perpetual grifter snake oil salesmen types aren't intelligent.
> All it can do is get exposed to specific examples, and learn that we don't like that.
I've heard it said that prison rehabilitation programs for prisoners diagnosed with psychopathy that are based around exposing them empathy for the victim are counter-productive. Apparently programs that teach these people to think about the consequences of their actions and how they're detrimental to their own personal well-being lower recidivism rates in this particular kind of group.
> The implication of what you're saying is that pathological liars and perpetual grifter snake oil salesmen types aren't intelligent.
No, I don't think that is the implication. I think you're making the "if all x's are y's, all y's are x's" mistake. I am saying LLMs cannot be moral because they don't have a mind, actual intelligence, or the ability to experience consequences. That doesn't mean that anything immoral is unintelligent.
the implication is drive and impulse to behave a certain way doesn't come from "intelligence"