Comment by alevskaya
1 month ago
I was a genetic engineer for ~20 years and have worked on frontier LLMs for the last 8. I used to engineer viral vectors and studied how to evade human immune systems for gene therapies...
The biorisk scenarios that the AI safety folks flog are fever-dreamed fantasies that have only the most tenuous connection to biological reality. As someone who cares about the real bio-risks of natural pathogens, I get pretty tired of fear-based marketing pretending that AI is a bigger threat than, say, animal agriculture.
It has always been ridiculous to suppose that some organization spent $500m on a microbiology / genetics lab, but ran out of money for scientists, so they have to ask Claude what to do. Or OTOH to suppose a guy in his garage set up a weapons-grade CRISPR lab without anybody noticing.
> The biorisk scenarios that the AI safety folks flog are fever-dreamed fantasies that have only the most tenuous connection to biological reality
Do you not think that at the rate ai intelligence and ability is increasing, this could realistically change one day soon?
Even if AI gets super smart, it will run into the limits of what we know about biology. Someone will have to do lab experiments to provide more knowledge to the AI. This is different from say building a computer virus or hacking since the AI can do all of these on it's own
> The biorisk scenarios that the AI safety folks flog are fever-dreamed fantasies that have only the most tenuous connection to biological reality.
As an expert, could you also provide your arguments please?
Anyone can write out the code for a bad virus. You can go download it from an open repository. It's only through deep interface with the world that the idea for the bad virus turns into an actual bad virus. The fever dreamers will say the LLMs will help you interface with reality to do the bad thing™ which their model let's you do. But you still need thousands to millions of times the effort And once made how do you deliver it in a way that might further your (bad) objectives? Presumably it just makes humans sick. There aren't "targeted" bioweapons, and among humans we are too damn similar for there ever to be. And all of this said, there is almost nothing special about the LLMs' abilities in biology. They only know what we know. They're not being trained autonomously with RL and a robotic wetlab. When that's a thing I'll start to take claims of biology risk more seriously. Right now they have the same logic as the paperclip theory of superintelligence risk. And cynically, it would seem that Anthropic purchased a biotech company and almost immediately decided to lock down biology work with their models.
> They're not being trained autonomously with RL and a robotic wetlab. When that's a thing I'll start to take claims of biology risk more seriously
So if IIUC your point is "they're not good enough at biology right now because they're not trained on it so they're not a threat".
To which I want to answer: "they're not a threat now but I see *no* reason for models not to be trained on biology pretty darn soon unless people like you convince the world otherwise."
Thoughts?
5 replies →
If you are optimistic about this given your expertise, I am very glad to hear it. As someone scientific but with little background in biology, I've been concerned about the bioweapons angle for a long time (and not because AI companies tell me to worry about it). Can you please explain a bit more about why you are not concerned? Between standard bioweapons like sarin and gain-of-function research on viruses, it's not obviously implausible to me that LLMs a few generations from now won't be able to guide a determined layperson through the steps needed to make something very destructive.
I think I'm less bothered by the risk because I've actually used these systems to do biological research, to work in bioinformatics and to work in the lab. And while I think that the leverage you get in bioinformatics is very significant, the laboratory work has never felt the same.
At best, you get much, much better ability to understand existing literature. the model itself has a very poor understanding of the physical world and that's masked by its knowledge of things people write about the physical world but it intrinsically doesn't have the same kinds of intuition and perspective that are really required to drive integration and completion in this space.
Is AI an important new tool in biology? Well, yes. Does it cause so much uplift in capacity that some rogue actor without biological research background could somehow destroy the world with a super-bio-nuke? I don't think so. I think that's just as logical a conclusion as the idea that next year one of the new frontier models will be told to make as many paperclips as possible and accidentally boil lake Michigan in pursuit of its goal.
> fever-dreamed fantasies
Reminder that what local LLMs are achieving today is the "fever-dreamed fantasies" of 5 years ago.
People really need to internalize that we will eventually have the technology for giving everybody the equivalent of a world class scientist locked in their basement that is willing to do anything.
No one knows the timeline, but it's inevitable (barring societal collapse or some kind of legislation)