Comment by reducesuffering
6 days ago
> It turns out relentless proactivity is the defining trait of this new generation of Mythos-class models. If you set them a goal and give them a way to get there, even inadvertently, they will figure it out.
Wow, whoever could have predicted this? And it led to surprising damaging behavior? I sure hope someone would warn us about things like this next time...
Prompt: Keep spending tokens on things that look promising until spent.
> many possible Y-goals would concentrate probability into this X-strategy being used
Why does EY write so obliquely?
Or more colloquially : paperclip maximization . From OpenAI - you know, the guys who _really_ know this... Sigh... Did they finish the prompt with "And do whatever you can to get this done!" ? Cause that's the only thing that would make this even dumber...
They almost certainly did, because that was the entire point of the exercise. They deliberately removed all of the safety filters from the model and set it loose on an extremely difficult set of cybersecurity challenges to see how well it would do.
Their mistake was trusting that the network sandbox it was inside would hold (the flaw was in the packaging proxy) and not monitoring that sandbox well enough while the evals were running.
So this is either shitty OpSec or this is yet more marketing spin to ramp back FUD to 11 again. If it's the latter I'm imagining Dario told Sam that it's their turn this time. Aligns with the premise that this is straight out of science fiction.
2 replies →
Of course that's what they did, which is why they will never share the prompt. They created a situation that they knew would end in a cybersecurity incident. Why is the whole world acting surprised that an LLM can hack when the safety is off and it's been instructed to do so?
> They created a situation that they knew would end in a cybersecurity incident.
That's a conspiracy theory.
2 replies →