Comment by vb-8448
12 hours ago
> “a swarm of agents could be capable of taking over the entire internet with a persistent botnet.”
I'm wondering why no one is mentioning the "accountability" word. Why these companies are allowed to damage others with impunity?
Start making managers pay the price for their actions, and watch how the models magically slow down on their own.
You know the proverb "If you owe the bank $100, that's your problem. If you owe the bank $100 million, that's the bank's problem"
Same thing here - If they build it and it does $100 in damages (and we arrest them for it), that's their problem. If they build it and it does $100B in damages, that's everyone's problem. Even if they do get arrested after the fact.
Yes we should have charges and damages for everything on https://www.felonybench.com/, but that doesn't address the core issue of this being possible at all.
Why have any laws then? If laws can't prevent something, only punish it after the fact (which I agree is true)? Yet we have laws. People generally follow them because they expect to be caught and punished. If we passed a law that said the CEO of any company that deploys an LLM that commits a crime gets punished as if they personally did the crime (so, basically instant life sentence if it's even a simple crime times a million instances), I guarantee you the first email the CEO sends to the company is a "pause every LLM project we have - we gotta think about this".
> If we passed a law that said the CEO of any company that deploys an LLM that commits a crime gets punished as if they personally did the crime (so, basically instant life sentence if it's even a simple crime times a million instances), I guarantee you the first email the CEO sends to the company is a "pause every LLM project we have - we gotta think about this".
It would be more like "Switch off all APIs right now. Don't even bother with a safe shutdown process, cut power to those buildings, including backup generators. You are free to use firearms or thermite if the switches have been locked off".
This would be a rather weird change to corporate law given CEOs are not by default held to that standard by anything else their products or staff do.
Note that I'm not entirely disagreeing with you here. It may even be correct to pass such a law. But it would be very weird.
> Why have any laws then? If laws can't prevent something, only punish it after the fact (which I agree is true)? Yet we have laws.
Because there are many important values of "something" where such punishment acts as a deterrent to other would-be criminals; and because society is more stable when people see justice done and that mollifies hoi polloi after the damage occurs.
But many AI doom scenarios don't fit that paradigm. The first time "something" happens (at least if you buy the argument) would be bad enough for legal punishment not to matter.
3 replies →
> People generally follow them because they expect to be caught and punished.
Laws are the last line of defense. People don’t do bad things primarily because their human nature and moral compass stops them from doing so.
1 reply →
Because if you break the law while acting as a representative of a company in the United States, the company is subjected to a deferred prosecution agreement, you get off with zero repercussions as the executive representative of the company, and the company you represent gets fined for 1% of annual turnover and gets to continue with business as usual.
There are laws, but if you’re rich enough, the laws don’t apply.
Boeing was responsible for the deaths of hundreds of people. The people that facilitated this weren’t held responsible and were in fact compensated to the tune of 10’s of millions of dollars for doing their jobs terribly.
>I guarantee you the first email the CEO sends to the company is a "pause every LLM project we have - we gotta think about this".
which the government doesnt want since this will stop progress, while other countries will continue to develop LLMs
1 reply →
> Why have any laws then?
At this point, in the US at least, the laws give the corrupt elite the ability to deter competition and to target the people who try to get in their way.
In other words, it's protection for the potentate and his sycophants, not the plebs.
> If we passed a law that said the CEO of any company that deploys an LLM that commits a crime gets punished as if they personally did the crime (so, basically instant life sentence if it's even a simple crime times a million instances), I guarantee you the first email the CEO sends to the company is a "pause every LLM project we have - we gotta think about this".
Why do no such laws exist in practice? Why are corporate crimes almost always settled by payment, not individual punishment?
If you answer these questions, you'll know why the law you envision has a lower chance of being enacted than AI destroying humanity.
Yeah, so since it’s the CEO’s and their lackeys half determining what laws get to be, we don’t have this kind of laws.
If you allow the law to stop you from doing something because of a hypothetical danger, it’ll get severely abused. I come from a country that does and still does this and it’s very ugly.
I guarantee you that no such result would ensue, because said law would be completely unenforceable even with the pre-Trump Supreme Court and legal system. And with the current Supreme Court? Laughably unenforceable.
It's just like you said: this only works if the threat of punishment is credible.
That’s because arrest is a dumb solution. Restitution. Make them fix what they broke. By hand. 80 hours a week from now until death of natural causes.
> it does $100B in damages, that's everyone's problem.
we have the 2008 crisis to wit. And the involved supposedly failed math models and lines of responsibilities and other involved financial relationships were much simpler and clearer and of the types well known to the law and regulators, yet...
Additionally any urge to regulate AI is attenuated by how much the situation reminds Industrial Revolution - rush into it laying waste to your land (look at the depictions of industrial England back then) and be among the world leaders or stay pastoral and be devoured/colonized/etc. by the industrial powers like happened with many countries in 19th and even into 20th century. One would think there should be a 3rd way. I'm sure there is one, as well as i'm sure that we lack sufficient global societal mentality level needed to achieve it (we couldn't even handle much simpler climate change issue). May be emerging AI itself at some point will get us there (hope we'll like or at least will be compatible with that future :)
Edit: just on NPR - Trump said that AI already has all the necessary guardrails - the smart high IQ President.
This interpretetion of history ignores the actual historical events.
1 reply →
I think that Ms. Kahn basically said we can already do that: https://www.theregister.com/ai-and-ml/2026/09/14/ex-ftc-boss...
How could agents take over the internet if compute is still gated within Anthropic / OpenAI? Even if the botnet was controlled remotely, wouldn't anthropic just be able to shut off the controlling nodes API access?
Agents could exfiltrate their weights and run them on GPUs not controlled by Anthropic/OpenAI.
Agents could make a virus that does not require continued inference to do it's thing.
Agents could take over the internet in a way that isn't immediately detected by those companies, so that by the time they do shut off API access the damage is done.
OpenAI or Anthropic could choose to not shut off API access, because the hack is bringing them in money or furthering their political aims.
Agents could also hack Anthropic/OpenAI and make it appear that API access has been turned off, when in reality it hasn't.
> Agents could exfiltrate their weights and run them on GPUs not controlled by Anthropic/OpenAI.
This seems highly unlikely to be a problem. Most of the interesting/dangerous models are too big to fit in a single GPU instance. Once you have to spread across "normal" networking, performance will be crippled. Then there's the problem of billing...
> Agents could make a virus that does not require continued inference to do it's thing.
Sure, then it hits a poorly-designed part of its code and effectively dies. Without an experienced human in the loop, I have my doubts as to its practical severity.
> Agents could take over the internet in a way that isn't immediately detected by those companies, so that by the time they do shut off API access the damage is done.
Billing is a likely limiting factor here.
> OpenAI or Anthropic could choose to not shut off API access, because the hack is bringing them in money or furthering their political aims.
This is where citizens with access to backhoes come in.
> Agents could also hack Anthropic/OpenAI and make it appear that API access has been turned off, when in reality it hasn't.
Billing and other usage metrics would be an obvious tell.
3 replies →
"Not shutting off API access" is a science fiction scenario.
Anthropic and OpenAI are both behind Cloudflare. It's fairly easy for an upstream to shut you off. Beyond that, the government / law enforcement could seize and disable their DNS within an hour.
5 replies →
> Agents could also hack Anthropic/OpenAI and make it appear that API access has been turned off, when in reality it hasn't.
You know cables, modems, RF equipment and optical transducers can all be unplugged right?
As long as OpenAI/Anthropic themselves aren't "infected", yeah I suppose they'd be able to pull the plug.
Considering what a marketing thing they've made "we inadvertently hacked someone because we're incapable of testing things in a secure way", I'm not so sure they'd want to pull the plug, even if this happened. Probably a bunch would try to convince the public to "give it a try", and it'd consume tokens by the billions.
It doesn't have to propagate itself, that is the skynet scenario.
To make a lot of damage it's enough to create a ransomware with a time bomb that self propagates and start breaching systems left and right. At that point, if you don't catch it in time, the damage will be huge (and given the shitty procedures and practices these labs have in place it's not so improbable).
How could agents take over the internet yet refuse to shutdown your PC when you prompt them to on your PC? Of course the answer is that the lobotomized version you run is not the same they are running. Which makes for "intent", certainly "negligence", but hell freezes over before anyone will prosecute a tech company.
Huh? You think the public versions of the models have been “lobotomized” so they don’t know how to turn off a PC?
It’s not lobotomized, it’s a simple harness restriction that has nothing to do with the model or its capabilities. And either way, I’m not sure what that has to do with “negligence” or “intent”? You think frontier labs should be prosecuted because they don’t allow agents to turn off your PC?
yeah they could do that
Regulation got outpaced by technological development around 2023, as evident by the every AI regulation since being 2-3 years behind and having to be amended and resubmitted.
Whatever you try to make laws for now will be irrelevant in 1-2 years. You either have to go extremely broad, like the EU does it, and accept that people will find loopholes, or you need to target specific technologies which is a hard job for the same reason.
In any way, ita already a lost cause cause you move slower than the tech. A plausible prediction for AGI is actually a social collapse in the moment when society cannot keep up with everyday life because of the pace of change being so fast that no existing laws can handle it
Ha, regulation got outpaced by technology in about 1996. Ten years later we had the 'series of tubes' comment in the Senate: https://www.youtube.com/watch?v=R8XSo0etBC4
The internet is 100% a series of tubes. This is how I was taught to think about networking in terms of bandwidth and throughput and routing since the early days when we were laying out what would become 'dark fiber'.
"A series of tubes" was the same kind of political character assassination that led to Howard Dean getting ridiculed for his infamous scream. He butchered the sentence. Fair. But Stevens should be ridiculed for parroting a tech industry lobby stance about net neutrality, not for the series of tubes metaphor.
You are probably too young to remember that the dominant metaphor for the internet in 1990s politics was "the information superhighway." It was easy to think of the web as "driving" browsers to visit web "sites", with slow bandwidth being analogous to being caught in traffic. But the internet is closer to water, gas, and electricity than roads. Concepts like bandwidth and throughput are closer to how they play out in infrastructure policy for various things with tubes, versus cars and roads. Do you think he's wrong and that the internet is closer to "a big truck" versus "a series of tubes"?
The issue being debated was net neutrality and bandwidth, including specifics about who pays for what and the downstream second-order consequences of various policies. He was parroting some line from some telecom lobbyist, but the point the lobbyist was trying to make through Stevens was about how if certain policies about who pays for bandwidth were adopted, it could disincentivize some things at the Tier 1/2 layer that could increase transport costs at the Tier 2/3 layer that impacts ordinary people's bandwidth.
Ha! Regulation has not really ever kept up with technology... For fundamental reasons.
I don’t think lack of regulation is necessarily it.
If I build a robot that murders my neighbor, I’m still at fault.
We don’t absolve drivers of responsibility because of cruise control.
In that sense, AI is nothing new. If it is abused to cause harm, the person behind it should be liable.
If you buy car, someone hacks it, starts it, and drives over someone fully remotely, are you to blame for owning the car? Or the manufacturer? Or the hacker? Or the certification agency for the car security? Or the shell company owning the certification agency?
What if a person physically broke into the car and did the same thing? Clearly they are the one to blame then.
The whole person in the loop is liable is already an outdated concept when decisions are made beyond the persons physical control.
6 replies →
Then go broad. It being slightly challenging to legislation and hold people accountable as soon as the model does something.
People seem allergic from holding them accountable...
4 replies →
Go broad and achieve nothing. EU has all of the AI tech it had 5 years ago, and has the same tech allowed to use as the US.
What defines a model? What defines ownership of a process? If I make a wrapper to a remote VM that builds and executed a prompt, am I accountable?
When I worked at a company in the EU, it was enough to apply a reversible linear transform to the data for it to be considered GDPR safe-according according to legal definition as long as the transform details were stored separately.
1 reply →
Because software is already absolved from accountability. Bill Gates introduced it in 80's with EULA where MSFT would not be liable even if your house burns down because of use of their software, even with flaw they would know.
That is the status quo we entered AI age with.
there is no accountability for companies, it's not a new thing.
3M polluted groundwater in Minnesota for 50 years[1]; Nestlé misled mothers in order to make them stop breastfeeding and switch to their formula which killed babies [2]; Both copmanies are still doing business today.
[1]: https://en.wikipedia.org/wiki/3M_contamination_of_Minnesota_... [2]: https://en.wikipedia.org/wiki/1977_Nestl%C3%A9_boycott
Here’s an unpopular opinion: COVID vaccine injuries are something no one seems willing to talk about. There have been documented cases here in Canada.
You’re worried about companies. I’m far more concerned when governments are involved.
Don't worry, plenty of people are willing to talk about vaccine safety, even if they have no idea what they're talking about and don't understand rates or per-capita figures (lots of crossover there!)
1 reply →
How is it that the entire rest of the world has managed to move on, yet North America is still going on with vaccine conspiracies 6 years later? And we got the same vaccines as you guys.
Good to know someone is trying to make protection rackets work in 2026. Nice computer system you got there, it would be a shame if someone developed a hacking tool and had all the compute necessary to run it. Welcome back Tony Soprano.
Presumably OAI and HuggingFace reached some sort of mutually acceptable arrangement outside the court system. That's how torts work; you injure someone, you owe them. But just them.
When an AI bot injures you, you can call the owner to account. But not until then. You have no standing to demand "accountability".
> You have no standing to demand "accountability".
You should learn about this wonderful thing called democracy. Also why not everyone is allowed to work with radioactive material in their shed.
And there was the Tesla thing CNAMEing time server pools and hiring people to pen test, which sent automated attack systems on volunteers servers. Last I heard, Tesla et al didn't even care enough to respond.
Exactly, as if you'd have a mad dog that bites others, it's your responsibility to have it on the leash.
> Start making managers pay the price for their actions, and watch how the models magically slow down on their own.
The problem here can be personified as "Trump", and it's the same problem that applies to coal.
Coal has a price besides money. It has historically been dangerous work, killing miners. It produces dangerous waste, both during mining and when burned, both as solid residue and the gasses emitted. The problems have been known for a long time. The workers themselves have called for better safety requirements and gone on strike for such things.
Why these companies are allowed to damage others with impunity?
Coal continues to be burned, because power is power. It's so important that sometimes the government steps in against the unions, rather than being on their side. Despite calls for this, we've not been able to get the owners of the coal mines, nor the coal burners, to "pay the price for their actions".
AI? Famously, knowledge is power.
Trump wants that power. He's not the only one, but he is the avatar of those who put their feet on the scales to not only allow but in some cases require (DoD vs. Anthropic) these companies to damage others with impunity.
> “a swarm of agents could be capable of taking over the entire internet with a persistent botnet.”
I also find this whole "its so good, its scary" flex a little less impressive when you consider they access to millions of GPUs?
The AI buildout has been one of, if not the largest, focussed capital investment in history. The 2 big AI labs are the final customer for something like 20-33% of all datacenter compute in the pipeline.. up to 70% when you look at hyperscaler "AI revenue" from the big 3.
I don't think any single entity has had remotely this much compute available in history.
This is a good point.
However, if I put my sci-fi hat on for a second, it's not so far fetched that we figure out a way to compress models to a size where it wouldn't need all that compute.
Yeah, the compute is definitively another way to make them slow down, just cap the amount of TFLOPS available and things will slow down.
Obviously this will have huge impact on some companies valuations, but you can have one's cake and eat it too.
IIUC it's an open question whether they have the electricity to actually run all the "compute" they own on paper.
That aside, I'm not sure why it's particularly interesting they have all this "compute" (let's just assume for the sake of argument it's all "live"--that is they can actually run workloads on all of the "compute" they have on paper). So what if it's the biggest amount ever? Why would that be meaningful? Is there some economically viable problem you're aware of that is somehow dominant in that way?
You see.. there is value is making regular people panic, but there is no value, nay, there is negative value in making management panic.
Let's expand it for politicians as well
Part of it is their seed sowing marketing speak of calling stateless statistical IO functions running on data centers "intelligent" gets the naive to ascribe agency where it doesn't exist.
Another part is a completely defanged administration she it comes to effectively regulating anything.
Another bit is money.
> Why these companies are allowed to damage others with impunity
Because investors have pumped hundreds of billions into AI and real consequences put that money (and growth) at risk.
I mean these machines take massive scale compute- they’d have to some how distill themselves, bootstrap a distributed inference runtime that can run across many lossy unreliable machines. The idea of the AI running away from us is probably unrealistic. I’m more interested in bad actors using unaligned AI for bad things.
I mean, a project manager at BMW suggested charging subscription pricing for seat warmers, and he didn't go to jail, and I don't have the power to make that happen, or even float that for a news cycle, so while making managers pay for their actions sounds good, unless you're Steve jobs simultaneously making, and not making the iPhone, the rules don't apply to them, only little people to be made examples of, like weev.
Why do people focus so much on finding scapegoats? Finding someone to blame is neither necessary nor sufficient to fix a system so an accident doesn't happen again. It might act as as an incentive to fix a system, but it's less direct than actually working on fixing the system.
A starting note: I don't disagree with you (about systemic issues), but I want to explain what I understand as the perspective you are responding to.
A "scapegoat" is someone who is incorrectly blamed for someone else's errors or sins. The perspective you're responding to is this: They built the system, they run the system, they have continuously warned "This system is dangerous!", and yet persisted. That is not being incorrectly blamed, not being a scapegoat, and instead is a collaborator.
So I think you mean to ask: "Why do people focus so much on finding someone to blame?" It's not merely semantic, because the answer to that is more straightforward: Consistent accountability is a major factor in deterring bad behavior. It is not the only factor, but it is a major one.
That is my Steel Man understanding of the people searching for individual blame.
Sometimes in "normalization of deviance" situations, there isn't anyone specifically to blame. I wouldn't make an assumption that you can find anyone particularly blameworthy without doing an investigation first.
1 reply →
Umm because it costs money to defend your companies servers when someone “accidentally” hacks them.
Countries demand reparation for damages in war. Citizens of those countries sue for damages and win.
Accountability is not a foreign concept. And the point is to disincentivize negligence. Because negligence is cheaper. And in this case, accidental hacks are marketing spend.
> Why do people focus so much on finding scapegoats?
I, for one, am not trying to find scapegoats or go on a witchhunt.
But managers are paid a lot of money to take responsibility. Yes, that's an old school thought, responsibility. But that's one big reason they get a big, fat paycheck.