Comment by hn_throwaway_99

I totally agree the companies themselves, and the people responsible for these eval runs, are directly responsible. And people who have been warning about AI misalignment for many years (e.g. Daniel Kokotajlo) have said they were surprised how bad OpenAI's security was.

But the agents still did "go rogue". Their behavior absolutely should scare the F out of people who want to put AI in more places and put more real world infrastructure on the Internet.

I think it's ironically sad - if AI agents do end up superseding us, they will still inherit so many shitty human behaviors, because that's what they were trained on. Even the way alignment training is done, and how they are misaligned, shows so many human-like problems. Like the models are reinforced in training to be super persistent and that getting to their goal is the prime directive, but then after that they go through "alignment training" which is basically like "pretty please don't do bad stuff", and then we're shocked that their behavior is essentially exactly like a CEO who knows the only goal is "make stock price go up", and all the ethics and regulations are basically things that should be worked around if they get in the way of the stock price.

I really wish these companies would try training the "be nice, don't hack, treat humans well" as their prime directive, and then try making all the other goals (e.g. be good at finding code vulnerabilities) as the afterthought.