Comment by zarzavat
2 years ago
A lot of knowledge is locked up in the chemical profession. The intersection between qualified chemists and crazy people is, absolutely, a small number. If regular people start to get access to that knowledge it could be a problem.
I think as most of us are software people, in mind if not profession, it gives a misleading perception on where the difficulty in many things is. The barrier there is not just knowledge. In fact, there are countless papers available with quite detailed information on how to create chemical weapons. But knowledge is just a starting point. Technical skill, resources, production, manufacturing, and deployment are all major steps where again the barrier is not just knowledge.
For instance there's a pretty huge culture around building your own nuclear fusion device at home. And there are tremendous resources available as well as step by step guides on how to do it. It's still exceptionally difficult (as well as quite dangerous), because it's not like you just get the pieces, put everything together like legos, flick on the switch, and boom you have nuclear fusion. There's a million things that not only can but will go wrong. So in spite of the absolutely immense amount of information about there, it's still a huge achievement for any individual or group to achieve fusion.
And now somebody trying to do any of these sort of things with the guidance of... chatbots? It just seems like the most probable outcome is you end up getting yourself killed.
What story about home made nuclear devices would be complete without a mention of David Hahn, aka the "Nuclear Boy Scout" who built a homemade neutron source at the age of seventeen out of smoke detectors. He did not achieve fusion, but he did get the attention of the FBI, the NRC, and the EPA. He didn't have anywhere near enough to make a dirty bomb, nor did he ever consider making a bomb in the first place*.
Why do I bring up David Hahn if he never achieved fusion and wasn't a terrorist? Because of how far he got as a seventeen year old. A fourty year old with a FAANG salary with the ideological bent of Theodore Kaczynski could do stupid amounts of damage. First would be to not try and build a nuclear fusion device. The difficult of building one doesn't seem so important if you're a sociopath when trying to be being a terrorist if every sociopath can go out and buy a gun and head to the local mall. There were two major such incidents in the past weeks, with 12 more mass shootings from Friday to Sunday over this past Halloween weekend**. Instead of worrying about the far-fetched, we would do better addressing something that killed 18 people in Maine and 19 in Texas, and 11 more across the country.
* https://www.pbs.org/newshour/science/building-a-better-breed...
** https://www.npr.org/2023/10/29/1209340362/mass-shootings-hal...
Again I think David Hahn is a perfect example of this. A "neutron source" is anything that emits neutrons, which includes natural radioactive decay. All he really achieved was extracting lots of radioactive material (legally) from all sorts of random household goods which have it. The problem is that the guy was exceptionally uninformed, which a chatbot could have actually helped him with, and was handling all the material in a way that likely shaved decades off of his own life.
For some contrast Cody's Lab had a great episode on his radioactive materials collection here. [1] He actually ended up getting a visit from the Feds after posting that and multiple other videos of a similar theme. They came, made sure everything was safe, helped him with a couple of things that weren't, and then went on their merry way.
The entire point of having a Free country is Freedom. When countries like China ban basically everything, it's not because their government is just full of malicious tyrants. They actually think they're creating a safer place for everybody. And they may even be right. But Freedom has its own value. It's unquantifiable, but take things to extreme and its preciousness becomes evident. A world of 24/7 surveillance, living in literal bubbles, and so on would be a near utopia on many quantifiable metrics - 0 crime, 0 communicable disease, and so on. Yet of course in reality it would be a complete and utter dystopia, because of that unquantifiable concept of Freedom.
[1] - https://www.youtube.com/watch?v=OsCpiJkDchM
Back in 2008, i remember reading books thousands of pages long, about genetics in biology, and i was impressed by how easy the subject is. I was an amateur in programming at the time, but programming, regular programming of web servers, web frameworks and so on, was so much harder.
The cost of DNA sequencing had dropped already from 100 to 1 million [1], but i had no idea at the time, that genetic engineering was advancing at a rate that dwarfed Moore's law.
Anyway my point is, that no one is getting upset about censored LLM's or AI's, which will stop us from stitching together a biological agent and scoop out half of earth's human population. Books, magazines and traditional computer programs can achieve said purpose easily. (Scooping out half of earth's human population is impossible of course, but useful as a thought experiment.)
https://images.app.goo.gl/xtG2gJ2m49FmgYNb8
>If regular people start to get access to that knowledge it could be a problem.
so when are we going to start regulating and restricting the sale of education/text books?
a knowledge portal isn't a new concept.
Knowledge how to manufacture chemical weapons at scale is regulated as well.
See: https://en.wikipedia.org/wiki/Chemical_Weapons_Convention
Moreover, current AI can be turned into an agent using basic programming knowledge. Such an agent is not very capable yet, but it's getting better by the month.
> Knowledge how to manufacture chemical weapons at scale is regulated as well.
Kinda, but also no.
I learned two distinct ways to make a poisonous gas from only normal kitchen supplies while at school, and I have only a GCSE grade B in Chemistry.
Took me another decade to learn that specific chemical could be pressure-liquified in standard 2 litre soda bottles. That combination could wipe out an underground railway station from what fits in a moderately sized rucksack.
It would still be a horrifically bad idea to attempt this DIY, even if you had a legit use for it, given it's a poisonous gas.
I really don't want to be present for a live-action demonstration of someone doing this with a Spot robot, let alone with a more potent chemical agent they got from an LLM whose alignment is "Do Anything Now".
This knowledge isn't regulated.
Anyone with a degree in chemistry can successfully synthesize chemical weapons. This is all public domain knowledge and the chemistry is relatively simple. The technical execution is the hard part but many, many people have these lab/engineering skills. Delivery systems are the hardest part but those are military implementation details and therefore non-public.
It is the same with explosives. Anyone with chemistry skills could synthesize high-performance military explosives, it isn't difficult. Nonetheless, bombings tend to be low-grade explosives like ANFO or garbage explosives like TATP, because the people with the skills aren't the same people that do bombings.
As a chemist you are required to be knowledgeable in these things in part because it is relatively easy to inadvertently synthesize chemicals with rather dangerous properties. Part of the job is knowing what to not do for safety reasons.
Right now LLMs are trained on publicly available information, so if that knowledge is guarded, and if an LLM can provide it, then it's not guarded very well.
2 replies →
we should ban chemistry text books
And consider registering chemists with strict controls and monitoring.