Comment by datsci_est_2015
10 hours ago
Digression, but this is the real Great Filter imo, not AI. I think technology advances to a point where it only takes one or two bad actors to type the right prompt to get a recipe for civilization-destroying bioweapons before you get anywhere near true AGI or anything relevant to the Kardashev scale. Biology is fragile.
But not that that’s a good justification for hamstrung models. I think it’s just the inevitable endgame and it’s more sad than scary
I have the same concern. If it becomes possible to engineer Captain Trips with a budget in the low 8 digits it won’t really matter what else happens.
I don't fully understand the instinct to regulate local models for this? It seems like the wrong place to address the problem.
You can download Ebola sequences right now if you want to. That's not the same as having an isolate. The difference is a lot of messy reality. This kind of work is not generally "one shot" (Claude make me a supervirus, make no mistakes), it requires lab space, iteration, and specific resources. It has a footprint.
Wouldn't it make more sense to monitor / regulate facilities where you can sequence or request assembly of DNA, RNA, restrict and monitor the supply of key reagents and so on?
>I don't fully understand the instinct to regulate local models for this? It seems like the wrong place to address the problem.
Those with such instinct deserve not the liberties they wish to deprive from others. Sam Altman and Dario Amodei do not deserve to touch their own uncensored SOTA models. Their access should be audited publicly to ensure this restriction holds, until such time they publicly renounce their censorship instincts and publicly release their uncensored models. Fair 's fair! :^)
Yeah I was thinking more on the scale of nation-state level actors. Most leaders would not consider a bio weapon due to potential blowback on their own population, as they generally don’t want to be the leader of a nation of corpses and ashes.
But if they don’t care, or if they already are the leader of a nation of corpses and ashes, maybe there’s not so much holding them back.
In my layman’s perspective, bio weapons seem to be more dangerous and volatile than nuclear weapons. And increasingly easier to engineer as biotech advances. I think about the headlines of “microscopic robots that target tumors from within your body” and see the horrifying opposite side of the double-edged sword.
It depends how easy it is (now, or in the near future) to turn information into weapons, and how realistic control of materials is. There's a reason we control access to plutonium, but information about metastable hafnium.
It does seem to me that for this specific problem the materials are a lot more amenable to control than the information is?
There's also this weird revealed threat model thing going on? Like why does it make sense to support heavy LLM restrictions but leave benchtop oligo synthesisers completely unregulated? (Note: I do agree that wanting to regulate BOTH is at least a consistent and defensible position).
I find it philosophically interesting because the problem is not strictly information control. Local models don't have any special extra information with respect to biological research. What has to be restricted is using information that's already publicly known in the wrong ways.