Comment by jswelker

> AI democratized the knowledge needed to build bioweapons. Like how with a 3d printer anyone can build a gun with no expertise.

Building simple guns out of pipes never was hard. Jury is still out if it is more or less work than getting a 3d printer working.

It turns out that LLMs, especially local LLMs, tend to hallucinate a lot when thinking about anything that's overtly fiddly or technical. This is even more the case when they're in a domain that isn't a natural part of their training data. If you have to "jailbreak" the model to get it to talk, you're so wildly out of the expected distribution that you'd be crazy to trust anything it says. It's basically making up stuff as it goes along. These are foundational issues with how the models are created, not something that a bad actor can just hack around.

(The biggest real safety issue in this kind of space is actually that the model might actively goad some unsuspecting victim into doing something incredibly dumb and dangerous to themselves as much as possibly others.

IIRC, there were reports of something vaguely similar happening IRL but involving casual mischief, not any kind of extreme attacks. And because nobody else seems to have managed to elicit the same actively goading verbiage from the model, it's implicitly suspected that the person involved was the one who introduced the problematic scenarios to begin with.)

You haven't made any argument either, so I suppose we're even.

> As a thought experiment imagine you have a biochemical PhD expert on the phone with you giving you step by step instructions on how to grow some dangerous biotoxin, giving you feedback in real time and helping you troubleshoot. Would you be more successful with this expert than you'd be on your own?

I think there is some difference of degree, but not of kind. A determined terrorist can relatively easily find many ways to kill people en masse today, no AI needed. The bottleneck is usually the actual physical execution in the real world, not theoretical knowledge.

not an expert but because i assume you need the physical resources to do it in the first place?

you don't even need to make it a thought experiment, we have real world cases of this. Anthrax cultivation is easy enough that a first year biology student can do it, but it's in practice so dangerous and difficult to aerosolize without the correct equipment that nobody does it. The information to easily produce many bioweapons is online, the internet and instant global communication already democratized knowledge, why are terrorists still driving trucks into crowds?

Plans for 3d printed weapons are publicly available, same argument was made then, if everyone can download a blueprint for a gun, are we all going to get gunned down? Hasn't happened. There's two errors in this Bill Gates argument. One is thinking terrorists don't already have PhDs, secondly overrating intelligence because that's the only thing they're good at, and when they think of terrorists they just think of their own inflated egos, but turned evil. Which isn't how real terrorists operate