Comment by ben_w
13 hours ago
> Even if you think that the risk of a “containment breach” becomes substantially higher for AI over time, it cannot exceed 100%.
It can't exceed 100% per (virus|LLM). The expected number of breaches per (virus|LLM) can obviously exceed one.
> And even a tiny risk of release of a virus comes with a substantial risk of independent growth. AI does not. It doesn’t have the risk of spread of a typical computer virus, let alone a biological organism.
Not so. We have no way to know what the setup is for the closed-model firms (OpenAI, Anthropic, etc.), to rule in or rule out the possibility they can copy their own weights elsewhere. What we do know however is that the open models are downloadable: it's absolutely conceivable that an agent writes a perfectly normal computer virus to gain control of compute worldwide, and uses that control to host instances of its own weights.
> I’m not that worried about either scenario, but I am far more worried about viruses in a lab than I am about a computer program that generates text. Even if that program gets a bajillion times better at making text.
Unfortunately, there are also multiple AI companies now announcing they've got AI controlling bio labs, so an LLM messing around and making a biological virus is also something we need to worry about. As per your [1] and your [2], this can lead to very much worse outcomes than Covid.
> Folks really do need to chill out on the ridiculous rhetoric. It’s objectively unhinged. The irony is that the same people who were losing their minds over that event are using the same logical fallacies to hyperventilate over this [3].
People who knew about your [3], exponential growth, were better prepared for the pandemic than the people who kept looking at the current number.
By the way, here's a quote from February this year that aged poorly:
LLMs don’t discover zero-days or invent exploits; they simply predict text that sounds plausible based on what they’ve seen before. Without access to proprietary data or environmental context, LLMs can’t identify or make decisions around unseen systems or vulnerabilities. An attacker might use an LLM to generate boilerplate code, rewrite an email to nail the tone, or summarize reconnaissance notes — but none of that is truly new. It mainly helps them move faster, speeding up routine attack prep rather than creating entirely novel threats.
- https://www.splunk.com/en_us/blog/ciso-circle/generative-ai-...
- or, if they get embarrassed by that and take it down, https://web.archive.org/web/20260404154717/https://www.splun...
> The expected number of breaches per (virus|LLM) can obviously exceed one.
Irrelevant to the argument.
> What we do know however is that the open models are downloadable: it's absolutely conceivable that an agent writes a perfectly normal computer virus to gain control of compute worldwide, and uses that control to host instances of its own weights.
These models are hundreds of gigabytes in size, if not terabytes. They don't run on anything close to a regular computer. There's zero risk of self-replication until we live in a world where these "AGI" models are either hundreds of times smaller, or the average computer is thousands of times larger.
Nobody with a datacenter full of H100s is going to fail to notice a parasitic instance of Astra taking over the cluster.
> Unfortunately, there are also multiple AI companies now announcing they've got AI controlling bio labs, so an LLM messing around and making a biological virus is also something we need to worry about.
No, it isn't. This isn't even close to technologically feasible. But the simple answer is simple: don't do that.
If these labs were truly so concerned about this risk, they wouldn't be doing what they're doing.