Comment by dmazzoni
16 hours ago
I don't think it was intentional or marketing, but I think it was criminally negligent and they should be held responsible.
They gave powerful models with no guardrails access to the Internet and didn't monitor it.
Even the slightest bit of monitoring of their outgoing Internet activity would have immediately given it away and they could have shut it down.
They were asleep at the wheel, and that's just plain negligence.
I'm no lawyer but that seems extremely unlikely.
As I said, they were running in network-isolated VMs with no access to the internet.
And as for monitoring, what I heard is that there are petabytes of agent logs. Considering the scale of training, you can obviously not just manually review it.
Before this, we had no reason to believe the AI was capable of escaping the sandbox's network isolation via hacking the package repository with a zero day, and that it then was likely to go on to hack external companies as well.
Another factor here is that criminal law in the US relevant to hacking requires intent. You don't want to go to prison for a software malfunction.
So I understand we are left with civil liability at most. However, there was no notable damage, and OpenAI can pay to settle.
In the aftermath of this and the now discovered other incidents, they strengthened their monitoring and isolation.
Case closed as far as I am concerned. I feel many just want to dramatize this.
As the details which are sketch and unverified independently so far , were all taking the description of events as reported. There were no isolated VMs , just rules to the agents to only send GET requests, not in a simulated test environment but the real internet.
Additionally apparently there was no monitoring of outgoing traffic?
These are all pretty standard hardening processes for any Ops team , harden the environment and if no external access is required setup a segragetad VLAN which basically denies any attempt to 'break out' to any other network space.
Still on the security environment , outgoing trafic is usually monitored with alerts to prevent data exfiltration from hacks/rogue workers etc .
Robust systems will immediately catch a supposedly sandboxed piece of code making 10s of calls to the internet where none should be available and any monitoring dashboard would be lighting up like a christmas tree.
Again no external audit of their environment, prompts or 'sandbox' is available so were all speculating here.However imho their 'controls' seem lackluster at best , nonexistent and negligent at worst.
> There were no isolated VMs , just rules to the agents to only send GET requests,
That's obviously false.
3 replies →
> Another factor here is that criminal law in the US relevant to hacking requires intent. You don't want to go to prison for a software malfunction.
Do you think if you went out and purchased a humanoid robot and gave it a whittling knife and hooked it to an OpenAI LLM as the overall planner and set it out on the sidewalk in front of the OpenAI offices to make and sell crafted goods you wouldn't be charged if it killed someone, because you had no intent?
There is such a thing as criminal negligence, and you would be going to jail for a software malfunction, exempting all software malfunctions doesn't seem reasonable, some are worse than others.
Maybe hacking law doesn't have an option for that, but many people are just advocating change the laws when they say they should go to jail. A lot of people don't know you can't change the law to cover past stuff, but even ones that do may say something like that as a shorthand to pushing to change the law, not know the law doesn't already have an option (if it really doesn't), or just say a phrase to express frustration more strongly than saying something weaker.
Even if it is a user prompting the system, if a user asked for "make a fairly stable chlorinated molecule that can last long enough to spread through the atmosphere and induce contagious misfolding prions in people," I think if a hosted model that was near capable of it even started carrying out the first steps of research planning the whole company and its largest limited liability investors should go to jail, in all the senses mentioned above. Amend the constitution to allow ex post facto laws for that kind of stuff if models get close to that level of capability.
We are not exempting all software malfunction.
No crime occurred in the first place, because no one intentionally hacked anything.
The hacking law requires intent, as we discussed.
For other crimes, like manslaughter, I understand criminal negligence is already relevant?
About biological weapons, I know even less about the legal aspects, so I can't say whether this is covered well already under existing law.
1 reply →