Comment by poopingatwork
3 hours ago
It's security research. You should want to know about all avenues of exploit regardless of intent. It's the only way to know risk.
3 hours ago
It's security research. You should want to know about all avenues of exploit regardless of intent. It's the only way to know risk.
We should maybe apply the same kinds of physical controls to this software security research as we do to gain of function biological research.
I.e. do not do these frontier model training runs on virtual machines in the cloud which depend on software systems to prevent internet access. Do them in a special data center that is physically disconnected from the outside world. Like a data center in a faraday cage under a mountain powered by a generator with no cables coming in or out.
Sure, but in order to know security standards you have to test and validate. Everyone accepts some standard as 'secure' after rigorous testing and validation.
I think we're learning the standards for sandboxxing change when you consider AI's ease in discovering exploits.
A sandbox good enough to contain human generated code and intent, is likely not good enough for AI generated code and intent (seems obvious to say that but).
And so poking on these issues, regardless of who asked to escape the box, is still worthy research.
How do they ensure all comments speaking about the exploit stay out of next train run?
Do you apply this same logic to GoF research increasing contagiousness or lethality of viruses? Is there any limit to this argument, and if so how do we define it?
Most virologists are not terrified of gain-of-function research the way the public is...
- BSL4 is really, really stringent, and these requirements and review processes are strictly regulated across the board
- Most modifications researchers are actually capable of, nature is capable of producing something way scarier. Researchers are making tiny tweaks in comparison to the zoonotic exposure in everyday life that has brought us viruses that do things we never imagined
- Most gain of function tweaks weaken some other aspect, for example the way contagiousness and lethality are at odds
- It's considered rather unlikely that COVID-19 came about via gain of function research, despite how much the public wants this to be true
- It is quite useful for research, a lot is learned by doing it
I didn't ask "are most virologists terrified of gain-of-function research the way the public is"
I'm not even necessarily against GoF research, and I mostly agree with all of your points.
I'm asking about GP's logic.
Insurers would disagree. You don’t need to know mechanism to understand / model risk outcomes.