The "dual use" paper this is commenting on is the clickbait equivalent of "encryption is for pedos", and maybe Derek's "not too surprising" is code for "Science editors are not discerning enough".
Like, this is the whole point of pharmacology: Predicting the biological interactions of chemicals (what they do to biological targets, how potent), and their ancillary physical properties (solubility, volatility, stability etc). For example,
Optimizing for mu-opioid agonist activity gives you super potent painkillers, drugs of abuse, and that stuff Russia gassed a theater with to knock out / kill hostages and kidnappers (i.e. fentanyl analogues)
Optimizing for inhibition of various proteases might give you chemotherapy drugs with nasty side effects, or stuff with nasty side effects and no known therapeutic use (i.e. ricin)
Optimizing for acetylcholinesterase inhibitor activity will turn up nasty poisons which could be purposed as "nerve agents" or "pesticides"
Optimizing for 5HT2a activity will give compounds that are great for mapping receptor locations in brains, which are also drugs of abuse, and which are also lethal to humans in small doses.
And the "predicted compounds not included in the training set" thing is just table stakes for any predictive model!
I mean, anything can be toxic with enough dose, but the b-subtype agonists seem a lot more toxic than the a-subtype agonists.
(Fun fact: 6-APB, a “research chemical” recreational substance became an actual research chemical because of it had better 5HT2b selectivity than what was previously used in lab research)
Was thinking of halogenated NBOMe series - observed in humans to have a pretty narrow therapeutic index re: death, cheapish synth, can be vaporized
But yeah 2b could be worse. Or many other targets as well
Funny thing, optimizing "research chemicals" for (1) uncontrolled synthetic pathway and (2) potency is common to Institutional, Druggie, and Terrorist researchers. None of them want to go through the bureaucracy for controlled substances and potency is good for [better controlled experiments / smaller quantities to transport / more killing power]
Fortunately we don't see any real work in the chemical and biological weapons space anymore. While it would still be pretty handy for terrorist groups, in actual warfare chemical weapons aren't super useful. See https://acoup.blog/2020/03/20/collections-why-dont-we-use-ch... .
On a tangent: it occurred to me recently that we also don't see much use of ICBMs with non-nuclear payloads, despite these being a fairly-obvious "dominant strategy" for warfare — and one that isn't banned by any global treaties.
I'm guessing the problem with these is that, in practice, a country can't use any weapons system that could potentially be used to "safely" deliver a nuclear payload (i.e. to deliver one far-enough away that the attacking country would not, itself, be affected by the fallout) without other countries' anti-nuke defenses activating. After all, you could always say you're shooting ICBMs full of regular explosive payloads, but then slip a nuke in. There is no honor in realpolitik.
So, because of this game-theoretic equilibrium, any use of the stratosphere for ballistic weapons delivery is effectively forbidden — even though nobody's explicitly asking for it to be.
It's interesting to consider how much scarier war could be right now, if we hadn't invented nuclear weapons... random missiles just dropping down from the sky for precision strikes, in countries whose borders have never even been penetrated.
> despite these being a fairly-obvious "dominant strategy" for warfare
I don't think these are quite as viable as you think. ICBMs are expensive. Probably tens of millions of dollars each, for a single-use item. Cruise missiles cost $1-$2 million to deliver the same payload and have a better chance of surprising the enemy.
ICBMs have longer range, but how often do you need to strike targets more than 1000km past the front line? They're inherently strategic weapons.
Conventional Prompt Global Strike is intended to provide the ability to deliver a conventional kinetic attack anywhere in the world within an hour. It has been an active area of weapons research for the US for 20 years. As you speculate, misinterpretation of the launch is a concern. [1]
As opensocket points out, there are many shorter range conventional weapons used across borders. The cruise missiles of gulf war 1 or the drones of the post September 11 world.
To tie this in with current events this is exactly what makes the no-fly zone idea in Ukraine so dangerous. All of the things you have to do to establish a no-fly zone and take away the enemy's ability to fire into and effect your no-fly zone look the same as a prelude to an invasion or nuclear first strike. This is made worse by the fact that many of the weapons systems you would be using are dual use. Meaning they were designed to deliver conventional or nuclear weapons. It's a massive gamble that the actions won't be misinterpreted or used to justify moving up the escalation ladder.
There are quite a lot of shorter-range conventional systems. In practice you don't need that inter-continental range for most purposes. For some modern examples you have the Chinese DF-21 and the Russian Iskander system. And a lot of those systems are dual-use: capable of delivering both nuclear and conventional payloads. It's not totally clear what that will mean in a conflict between two nuclear powers. What do you do when early warning radar picks up a ballistic missile coming in when you can't tell if it is nuclear or conventional? Plus this isn't a video game, you won't hear some alarm going off after it detonates indicating it was a nuclear explosion. You'll need to send someone to do a damage assessment, and that takes time.
> It's interesting to consider how much scarier war could be right now, if we hadn't invented nuclear weapons... random missiles just dropping down from the sky for precision strikes, in countries whose borders have never even been penetrated.
Why are cruise missiles any less scary? They are indeed used in precision strikes across country borders, and can kill you just the same. The existence of nuclear weapons still allows some countries to use cruise missiles, as we see happen almost every year.
I'd consider how much more docile the nation's of the world would have become sans nuke.
If the possibility of an untraceable, space borne, hypersonic weapon was on the table we might have had a better deterrent than nuclear weapons. The lack of fallout and total deniability makes it almost certain they would have been deployed and quite concisely ended a few conflicts at the onset.
It is alarmingly frightening, moreso even, because the impact could be extremely precise- leaving infrastructure intact.
Shorter range ballistic missiles have been heavily used in multiple conflicts around the world for many years. The US military has been researching the possibility of using conventionally armed long range ballistic missiles to fulfill the prompt global strike mission. Potential target countries have no anti-nuke defenses. But there is a risk that Russia or China could misinterpret a launch as aimed at them.
in WW2, Germany used V2 missiles for indiscriminate bombing of cities (primarily London). I can imagine it would look like that, but worse - and having gone to a few museums that showed Blitzkrieg London, that was bad enough as it is.
To be fair, if these become a reality, they would likely strike targets in the Pacific Ocean and South China Seas, far away from the US, but the potential to spook nuclear nations is still there.
i would imagine delivering a conventional warhead with an ICBM has a very high risk of being mistaken for a nuclear armed ICBM. Also, they're expensive. Putting a JDAM package on an old iron bomb turning it into the most advanced precision guided munition is very cost effective.
Equilibriums change, every major US platform was at one point designed to be nuclear capable, i.e. cruise missiles now liberally launched from planes/bombers/ships that are all nuclear capable. There's no a reason nuclear countries who get attacked by any US platforms should assume any incoming ordinance ISN'T nuclear, down to gravity bombs, except for expectation - knowing US has overwhelming conventional capabilities and would rather use it than nukes.
Same will apply as conventional ICBM matures - we haven't seen much of it because ICBMs have not been sufficiently accurate unless carrying nukes where CEP in meters don't matter. For countries with power projection, it was dramatically cheaper to get closer first and deliver less expensive ordinance. Conventional ICBMs seem effectively forbidden because most actors assume they're too inaccurate for anything but nukes and too expensive for anything but nukes.
But that's changing - there are hints that PRC is pursuing rapid global strike i.e. US prompt global strike, because IMO it's the great equalizer in terms of conventional mutually assured destruction precisely because it isn't banned. A lot of articles being seeded on SCMP about PRC hypersonic developments that spells out meter level CEP ICBMs designed to conventionally attack strategic target of depth, aka Prompt Global Strike.
Ergo (IMO) PRC maintaining no first use nuclear policy while conducting massive nuclear build up to setup credible MAD deterrence. This sets up the game theory of accepting that conventional ICBM attacks on homeland from across the globe is possible and that it's best to wait for confirmation unless one wants to trigger nuclear MAD. Entire reason US / USSR and countries that could moved to Nuke triad or survivable nuke subs was because it bought more time than hair trigger / launch on warning posture.
This makes a lot of sense for PRC who doesn't have the carriers, strategic bombers or basing to hit CONUS (or much outside of 2nd island chain). It makes a lot of sense for any nation with enough resources for a ICBM rocket force but not enough for global force projection (basically everyone). World will be very different place if such capabilities proliferate. Imagine any medium size country with ability to hit stationary targets worldwide - fabs, server farms, power stations, carriers being repaired in a drydock.
I’m not sure there’s much difference between the Russians using air-launched cruise missiles (with ranges of hundreds to potentially thousands of kilometers and almost always capable of carrying a nuclear warhead) launched from their Tu-95 Bear strategic bombers (equivalent to the B-52), which Russia has done several times now in Ukraine.
> So, because of this game-theoretic equilibrium, any use of the stratosphere for ballistic weapons delivery is effectively forbidden — even though nobody's explicitly asking for it to be.
Interesting! SpaceX was hoping to one day use Starship for quick intercontinental flights. I wonder if this unspoken rule would make that prohibitive?
There could also be effective anti-missile defenses, which are currently politically impossible due to the fact that developing them puts you outside of the effectiveness of deterrence.
Chemical and biological weapons are very useful in warfare as a way to demonize one combatant. False or doubtful claims of chemical weapons deployments have an effect on the response of the public and international organizations that is entirely out of scale with the damage that could be inflicted.
Just doesn't seem like it's worth it even for terrorists.
Why invest a bunch of time and effort making more and more deadly poisons when we've already got a wide variety of them that are cheap to manufacture, well known in how they work, and don't cost a bunch of research money to uncover?
At the risk of putting myself on a watchlist, They already know that, so They have an eye on certain kinds of labware, different precursors, and such. And They already have antidotes to some of these poisons.
One could optimize for compounds with hard-to-monitor precursors. Compounds that can be transported with low vapor pressure and volatility, so they cannot be easily sniffed out.
Or imagine a lethal compound with a high delay factor. Or something with specifically panic-inducing effects, perhaps hemorrhagic with a side effect of your skin sliding off in great slick sheets. Another interesting high delay factor compound might induce psychosis: have fun tracking where these gibbering maniacs were a week ago.
With a sufficiently dark imagination, "needs" could be identified for all sorts of compounds.
Remember, the goal is to throw a monkey wrench into the gearwork of an opposing civilization, not necessarily to kill. Fear of the unknown is very effective for this.
True, though you have to remember that threat is a social construct and isn't necessarily a rational measure. The 2001 anthrax attacks killed 5 people, injuring 17, and shocked the nation. As a direct result Congress put billions into funding for new vaccines and drugs and bio-terrorism preparedness. If 5 people were killed and 17 wounded in a mass shooting by a terrorist, would we really have reacted as strongly?
If you wanted to install fear into a country, I think being attacked by some custom, previously unknown chemical weapon would scarier than sarin.
from what i understand the delivery of a chemical or biological weapon is the hard part. For most things, you can't just pour it out on the ground to have a huge effect. Somethings you certainly can, weapons grade Anthrax probably just needs a light breeze to devastate a city but something like that is beyond the reach of your average terrorist groups.
They are amazingly useful in real warfare. Drop nerve gas on a city, walk in a couple days later. WMDs are the only way to really take a country by force, and of all of them, chemical weapons are the most palatable and also the easiest to produce.
Considering this, defense against them is at least mildly important. A proper defense only exists by considering offense, so they're still developing chemical weapons somewhere. The modern hot topic is viruses and other pathogens.
That's not how it works, and they're not very useful at all. The amount of actual product you need is non-trivial and at that point you might as well just use modern conventional munitions.
The reason why it fell out of favour isn't because it's dangerous, it's because it was ineffective outside of TV and film.
They go against the Geneva Protocol, and it's not even allowed to stockpile them so not even useful if you are a terrorist with a death wish because then there are simpler ways to end your problems.
The real fun starts when somebody starts using techniques like this that overcome the weaknesses of known chemical weapons and provide specific advantages. It's also kind of hard to monitor computational chemical research.
It's my understanding that the Soviet army doctrine in the '70s and '80s included the use of chemical weapons. That hypothetical threat put a hell of a lot of friction on NATO in terms of training, supplies, and preparedness.
This will probably come in handy for industrial espionage type tasks.
Lets say you had a nation-state enemy who eats a lot of some ethnic ingredient. Come up with a cheap artificial flavor/color or process that is optimize to give heavy consumers cancer in 30 years. Not in one year, that will show up in the approval process. Then have an agent in the target country "discover" thru random chance this really excellent food dye or whatever.
Now you kill half the population with cancer, you're gonna get nuked in response, even non-nuke countries will be pissed off enough to get nukes just to nuke the perpetrator. But lets say you make the victims fat and sick and die a little younger just enough to get 1% hit on economic growth...
Some people would say this is how we ended up with trans-fats and margarine and vegetable oils in general or certain veg oils in specific.
Certainly, corn syrup has caused more human and economic devastation that fission, nerve agents, or most any WMD I can think of...
The problem with this is that this is basically genetic engineering, you might successfully make a low-level economic growth impact now, but future generations will be resistant to the poisons as those weak against them die off. You are securing your own demise long-term if you don't subject your population to the same.
If this were really a practical concern, machine learning would be designing drugs that fly through the clinic today. They aren't and so this paper, though click-grabbing, is probably of no practical consequence.
One reason is lack of data. Chemical data sets are extremely difficult to collect and as such tend to be siloed on creation. Synthesis of the target compounds and testing using uniform, validated protocols are non-trivial activities. They can only be undertaken by deep pockets. Those deep pockets are interested in return on investment. So, into the silo it goes. This might not always be the case, though.
For now, the paper does raise the question of the goals and ethics around machine learning research. But unintended and/or malevolent consequences of new discoveries have been a problem for a long time. Just ask Shelley.
A successful drug candidate must be useful in the treatment of human medical problems and not have harmful side effects that outweigh its benefits. A weaponized poison may have any number of harmful effects without diminishing its utility. A compound with really indiscriminate biochemical effects, like fluoroethyl fluoroacetate, makes a potent poison without any specific tuning for humans. It's much easier to discover compounds that genuinely harm people than those that genuinely help them.
"Now, keep in mind that we can't deliberately design our way to drugs so easily, so we won't be able to design horrible compounds in one shot, either. "
I would discount this, heavily and concerningly, as a false sense of security. The reality is that prohibitive factors in creating new drugs from compounds discovered similarly (by AI or other automated process) is almost entirely due to testing safety procedures and regulations... If the bad actors are trying to find the most lethal compound with no such oversight - and chances are very high that they aren't bound by any such regulation if they're state-level labs operating under impunity - there is nothing but the synthesis that would make the formulation and testing of these as impractical as the author claims. Take away the years-long, heavily scrutinized and regulated multi-stage billion-dollar path to drug approvals and you'll find that barrier is not so high.
I would like to think this data could be helpful to any organizations looking to proactively develop detectors or antidotes for such compounds - especially if the threat was previously unknown to them.
Let's say an entirely novel class of toxin was found in a cluster of these predictions that has no existing references in private or public records - it could be that another organization has discovered and synthesized something similar through one of many other paths.
Many lines are drawn between this type of approach and that of whitehat hackers. You must necessarily create the vulnerability to mitigate it. It feels like "white hat" biolabs claiming the same are operating on the same conundrum and that the difference between "studying for the sake of mitigating" and "creating a weapon" are fundamentally indistinguishable without an absolute knowledge of intent - such is impossible from the outside.
> The reality is that prohibitive factors in creating new drugs from compounds discovered similarly (by AI or other automated process) is almost entirely due to testing safety procedures and regulations
Most drug candidates fail because they don't work, not because of any regulatory procedure. About 50% of drug candidates that enter Phase III trials--the final clinical trial before approval--fail, and that's almost always because they failed to meet clinical endpoints (i.e., they don't do what they're supposed to do), and not because they're not safe (toxicity is Phase I trials).
That "not working" part has some nuance to it as well. How well do we predict ADME? Is there binding with some off target protein that makes it terrible? Maybe it just doesn't bind to the desired target at all.
Toxins don't have those constraints, its not even about regulation. Making something that's safe is way harder than making something that is not safe, purely because of the complexity involved in making the thing safe.
I was thinking this as well. If a new drug works well for 99% of the people, has mildish side effects for 0.9% and is really bad for 0.1% of people, that's no good. But if a nerve agent kills 99% of people and is not effective on 1%, that's just fine.
Also, compounds that are fatal tend to be so to all life forms vs just humans with the variation being dosage. It goes without saying the odds of taking a compound and finding a drug that does one very specific thing without doing anything else.. and demonstrating that it's safe in people, is orders of magnitude less likely than finding a compound that is lethal.
Hype over crap like this grinds my gears. Organophosphines like VX are ALL toxic. There's about a zillion such toxic molecules all containing the same functional group. This study does not demonstrate that this tool is better generator of toxic molecules than anything that includes the basic rules of valence and rudimentary understanding of shape similarity.
When thinking about whether ML does something novel, we must always compare with some simple alternative. I would be impressed if it'd predicted something like Palytoxin, a highly specific molecule with extraordinary toxic activity. There's no way the tools of this paper would though.
> ricin is (fortunately) not all that easy to turn into a weapon, and the people who try to do it are generally somewhat disconnected from reality (and also from technical proficiency).
Of course that fact was no barrier to much hype about the "dangers" it posed, either. I suspect the same now; that we have more to fear from the fear junkie propaganda than the actual facts.
I personally fear the lone-wolf attack drastically reducing in cost and effort. Where it would once be cost-prohibitive to design and manufacture your own nerve gas or lethal virus, these days with AI/ML and Crispr-cas and the like, it feels like any intelligent, deranged person wanting to take as many people to the grave with him has the tools to do just that.
Intelligent and deranged persons already have the tools to make way more casualties with way less effort using guns and/or explosives. The "problem" for them is that people who get sufficiently deranged to think that killing a few hundred (or even thousand) people will meaningfully solve the problem they are upset about will also be sufficiently deranged that their ability to reason coherently will be drastically reduced.
>>The thought had never previously struck us. We were vaguely aware of security concerns around work with pathogens or toxic chemicals, ...We have spent decades using computers and AI to improve human health—not to degrade it. We were naive in thinking about the potential misuse of our trade...
Of course now, the next step is to use the technology to preemptively search for and develop antidotes to the new potential weapons their tool has discovered.
We already (sort of) do this. AI/ML is probably used for simulating nuclear explosions, and is [arguably] even more useful and accurate than actually setting off a bomb, and measuring it.
It makes sense that it could be weaponized. When Skynet becomes self-aware, it would probably just design a chemical that kills us all, and would aerosol that into the atmosphere. No need for terminators, just big pesticide cans.
I'm quite sure we have already invented several chemicals that match your description -- like sarin gas, invented in 1938 by someone who, indeed, wanted to create a decent pesticide. A lethal dose of sarin gas is something like 28-35m³/min over 2 minutes exposure, according to wikipedia. [0]
Hitler was well aware of its creation, and I believe quite a lot of the stuff was produced for the purpose of warfare. There were several in the Nazi military who wanted to use it, but Hitler declined.
That seems rather odd, given his indifference to exterminating people with gas on an industrial scale beyond the theater of war. It has been suggested that Hitler was probably aware that to use sarin gas would be to invite the allies to do so in response, which would result in a dramatic loss of life on the German side due to the sheer lethality of such chemical weapons. [1]
Perhaps he thought it easier to stick to conventional warfare, in which the pace is more manageable than with WMDs, where you would start going down the road of mutually assured destruction but without the strategic framework in place to prevent anyone from actually wiping out a population before realising how bad an idea it would be.
And I think this reluctance to change the game, this seemingly deliberate moderation, perhaps best demonstrates the true difference between the machine and human in warfare.
It is not a difference in innovation -- we have always been very good at inventing highly optimised ways to end life.
The difference is that a machine intelligence will not hesitate. It will not ask for confirmation, pause or break a sweat. It will pull the trigger first, it will point the bombs at anything that is an adversary and anything that could theoretically be or become an adversary, and it will not miss. And it will not have to face ethical criticism and historical condemnation afterwards. [2]
[2]: Assuming this is a Skynet-like machine intelligence, which doesn't really have the capacity for remorse or negotiation and seems primarily, indeed solely occupied with the task of ending human life.
Obviously, a true AI that is essentially a conscious mind equivalent to our own minds, may experience the same hesitancy that most of us would, were our fingers to be over the buzzer.
Unless the AI independently arrives at a different set of values to us, like the Borg or something.
Chemical weapons are expensive. Consider the logistics and training required to effectively deploy them, plus any specialized equipment. Meanwhile, they're only useful as long as your opponent doesn't know you're planning to use chemical weapons, since countermeasures are relatively cheap and every major military knew what to do about them by the time WWII broke out. As soon as your enemy knows to beware chemical attacks, all you're doing is annoying them while making it hard for your own troops to advance (they have to put on chemical suits/masks themselves, or else wait for the gas to disperse). Very hard to use effectively in maneuver warfare. They didn't even prove very effective in WWI, which was much closer to an ideal environment for their use.
Its the classic logistics problem. Scaling ratio of weight vs volume or something like that. Just like nukes, if you heat an enemy soldier to 100M degrees he isn't any more dead than heating him to 10M degrees and volumes expand very slowly with mass so making bigger and bigger bombs is a fools-errand.
Same problem with chem weapons. You hit a tank brigade with 1000x lethal dose they aren't any deader than if you hit them with 1x dose. But if the bomb misses which is likely, all you've done is REALLY piss them off. Nerve gas in an empty wheat field just kills a bunch of corn bugs but it really pisses people off. If you target their tank brigade and miss, they'll target your home town, as we did to them with conventional bomb even without having been nerve gassed to start with. If you target their home town then the brigade you missed is going to be unhurt and really angry. Its the kind of weapon thats pretty useless unless you have infinite supply and infinite logistics. Like cold war USA or cold war Russia.
The allies had better logistics than the Germans so they knew the second time around in WWII that trying to go chem is just going to end up in the German's getting more chem'd than the allies.
Another issue is WWI and previous its all about siege warfare and breaking sieges where WMD is awesome and useful, whereas WWII and newer is all about maneuver warfare and blitzkrieg and all of Germany's plans and all of their early success were based on the idea that anything in range of shells or aircraft today is going to be occupied rear supply area next week at the latest, so destroying it would be pretty dumb because we need that area to be the rear of the battle space next week. For a modern comparison the USA could have nuked the green zone in Iraq and there's absolutely nothing anyone could have done about it, but 'we' knew we'd be occupying the green zone and needing something like the green zone, and the green zone is sitting there for the taking, so in an incredibly short term perspective it would have saved troops and saved time and saved effort to just nuke it instead of taking it the old fashioned way, but in medium and longer term it would be counterproductive to war efforts to use WMDs against the green zone, so we didn't.
Hitler probably also experienced gas (not sure his generals did, though). People forget that he was actually a decorated NCO, from WWI (which had a lot to do with his terrible attitude, later in life).
It was fairly worthless, militarily. High risk, big mess, no real tactical advantage, and it just pissed everyone off. Its only real efficacy would have been for bombing civilian targets, and I don't think they had the delivery mechanisms.
The article assumes that full development of a new chemical weapon would require more development effort. With regards to military usage: storable at room temperature, relatively easy to manufacture from commonly available precursor chemicals, etc. [1]
How true is that? Are there components of this process that make things easier now? Where I have chemical structure X, and a system generates the process steps and chemicals needed to produce X. How much of the domain in chemistry / chemical engineering has been automated these days? What are the future prospects for this?
[1] I assume one of the design goals for a new chemical weapon for military use is that it breaks down in the environment, but not too quickly (like say in a week or a month). Though I suppose if you want to just destroy civilization you would design for longevity in the environment instead. And being able to seep through many kinds of plastic if possible.
> Where I have chemical structure X, and a system generates the process steps and chemicals needed to produce X.
Undergraduate chemistry students spend a fair amount of time learning how to look at a novel structure X and by disconnecting "backwards" it into simpler components, deduce a route by which it might be synthesed "forward" in the laboratory from readily available starting materials.
There's an excellent book on this, "Organic Synthesis: The Disconnection Approach", by Stuart Warren.
Any other major topics or readings you could recommend for someone wanting a general understanding of key concepts in modern chemistry? I'd suppose generally: materials, synthesis, o-chem, and chem-eng.
My own background: began a hard-science degree. One year undergrad uni chem.
While it's worrying and worth thinking about, the track record of using AI to generate pharmaceuticals to do good has been "mixed", except really it's just been a bust. It may someday do great things, but not much yet, and one silver lining is that AI-generated toxins are unlikely to improve on the human-designed ones, either.
"That is, I'm not sure that anyone needs to deploy a new compound in order to wreak havoc - they can save themselves a lot of trouble by just making Sarin or VX, God help us."
Wow - I picked up on this earlier today, and even quoted the same as in this article. I was amazed that the scientists had not considered that AI could be/is being used for harm. (Was downvoted for this, but whatevs.)
It struck me as incredibly naive, but then - what would someone else do in their situation? Most of us work in silos without awareness of how our work is used, and I suspect we are often causing (unintentional) harm to others whether we are scientists, programmers, in finance, in health, in government, etc. If we realise our predicament, there isn't an moral authority to make things right. There is only the legislation that was been written by lobbyists paid by the corporations we work for.
Putting the article in broader context, perhaps it is about the creation of a moral framework for AI intended to pacify our disgust at the system we find. I expect that we will be expected to look away as AI "ethics" committees justify the unjustifiable, but call it ethical. As whatever-it-is is found to be ethical after all by ethical authorities, most of us we will wave this through and consider that we have acted judiciously. IMO.
Chemical weapons above a certain level, bio weapons and nucs are all seen as weapons of mass destruction and are not really that useful tactically. Introducing strategic-destabilizing elements to a conflict greatly increases its unpredictability and probably is a health risk for the leaders involved.
This is essentially what the pesticide and herbicide industries have been doing since their inception, i.e. designing molecules that efficiently kill animals, insects and plants. It seemed like a miracle at first, but the long-term consequences of things like persistent chlorinated aromatics and their derivatives (Agent Orange and dioxin for example) eventually appeared in human populations.
The development of the toxic nerve agents (organo-phosphate compouds mostly) in particular was a side effect of research into insect toxins. The nerve agents were discovered in this manner, they worked too well. Nevertheless, these pesticides were deemed safer than the organochlorines because they degraded fairly rapidly after application (although they are implicated in nerve damage related diseases like Parkinson's in agricultural areas).
Insect infestations are indeed a big issue in agriculture and can wipe out entire crops if not dealt with, but there are plenty of options that don't require applications of highly toxic or persistent chemicals.
Otherwise, this is just another of the many issues modern technology has created. Smallpox is another one - in the late 1990s, there was a great debate over whether to destroy the last smallpox samples - and then in the mid 2000's, someone demonstrated you could recreate smallpox by ordering the appropriate DNA sequences online and assembling them in a host cell. Then there's the past ten years of CRISPR and gain-of-function research with pathogenic viruses, a very contentious topic indeed, and still unresolved.
I hope recent events have illustrated that if it weren't for the people who develop lethal weapons, we (as in you and I) would be helpless against the bullies of the world. Unilateral pacifism is cute philosophy only when there are rough men standing ready to do violence on their behalf.
I strongly agree with this sentiment. However, it is hard for an ethical person to participate in developing war technology when possession and usage of the weapons is purely a political question, and history also has seen our side of geopolitics commit atrocities.
My stance has previously been that I am unwilling to work on weapons technology, because history has shown that these weapons sometimes end up being used for an indefensible cause. Then all of a sudden you're an accomplice to murder, and getting away with it.
In the light of Russia's invasion of Ukraine, which is just a continuation of its historical imperialism, working on weapons is something I would be perfectly okay with and probably even motivated to do. But stop for a moment and think what a history of aggressive military actions does to our society's ability to recruit for this important job.
The flaw in that logic is that, if it weren't for the people who develop lethal weapons for the bullies, we wouldn't have to fear the bullies.
Also, I think the design space of "radical defense" is under explored. Our (western) armies are still designed for attack and force projection, although we have long since renamed our war ministers secretaries of defense.
But I wonder if you could develop defense capability to make your country unattackable. Not by threat of retaliation, but for example by much much stronger missile defense. Or by educating ("indoctrinating") your own population, so that an occupier would not find a single collaborator? Or by mining your own infrastructure, and giving every citizen basic combat training (a bit like the swiss)? Or by fostering a world-wide political transformation that is designed to prevent wars from happening at all?
I think if we wanted to spend money researching stuff to keep us safe, it doesn't necessarily have to be offensive weapons.
Especially since we cannot really help Ukraine with anything but tech and they are outnumbered, so the only advantage we can give is how much better our weapons are than Russian ones.
Have you considered, though, that we (as in you and I) might be some of these bullies in the world and that these rough men aren't just standing by, but are actively doing violence on our behalf ? One needn't look much further past recent events to find examples aplenty.
I understand the point you are trying to make, but it's not as easy as pretending that the weapons "we" develop are purely for morally and ethically righteous purpose.
With respect you may be making some unfounded assumptions about what
I've built and what I believe.
My point was really about the fact that this job title "Lethality
Engineer" actually exists. And moreover, that it asked for medical
qualifications, which would go against any doctor's Hippocratic oath.
Most of us who've done defence related work are happy at the edges,
with tactical information systems, coms or guidance (my stuff ended up
in targeting).
But when it comes down to figuring out how fragments can be arranged
around a charge to make sure the waveshape optimally penetrates as
many nearby skulls as possible... hmmm suddenly not so gung-ho about
it.
That's not a distant, theoretical morality about tyrants and
bullies. I've no problems contemplating my family's military history
and am plenty proud of it, even though we'd all rather live in a world
without this stuff.
Until the lethal weapons are turned on those (countries, groups, people) who develop them. Kind of like gun owners are more likely to be harmed (or harm others) by their own guns, notwithstanding the arguments about personal protection used to justify such ownership.
This has already happened with groups the US has armed in the past. The US itself has been the bad guy sometimes.
There is no proper resolution to this struggle, and people who are guided by their conscience should not be attacked for having a "cute philosophy" that relies on "rough men standing ready to do violence on their behalf."
I generally agree with this, however, the rough men willing to do violence on our behalf are more and more becoming the quirky scientists who are very disconnected from the actual impact of their work. I think there's a big difference between those types of people. It seems like people don't feel the weight of violence as much as they used to. I imagine this will increase as we develop more AI driven weapons.
This comment is confusing two completely separate things. There's a world of difference between not being willing to defend yourself and actively trying to come up with more aggressive and lethal weapons.
The argument that "we need defense!" only justifies the need to stockpile and develop _sufficiently_ lethal and tactical weaponry to neutralize incoming threats (like anti-ballistic systems). It doesn't justify inventing deadlier weapons. No dispossessed victim of foreign invasion has ever needed a bioweapon to assert themselves, and there's no chance developing one would ever be used for anything but war crimes. You should absolutely turn down roles like "Lethality Engineer" from an ethical standpoint, even if you agree military defense is necessary.
People raise the spectre of deterrence as a utilitarian justification for needing more powerful weapons ("har har, they'll think twice about attacking us if they know we have nukes!"). But that's narrow thinking. Deterrence can be achieved in other less-damning ways, like strategic alliances and building more robust defense systems.
> Many years ago a colleague who works in defence told me about a job posting he'd seen but was having a moral struggle with.
This is a good struggle to have. What's ironic in many cases is that we don't experience these quandaries in other jobs, but the ethical and moral ramifications still exist. The early days of search in Google or social in Facebook probably didn't elicit the same kind thought process as a lethality engineering post. (Anecdotally I spoke some years ago with an acquaintance Googler who told me that he enjoyed working there precisely because he was working on privacy issues that worked against some of the advertising side of the business.)
I've worked in telecommunications, industrial systems engineering, and energy. There are ethical and moral issues in the work that I've done/do as a contributor in each of those domains, even though I'm not involved day-to-day in decision making that feels particularly moral.
One of the base assumptions we probably need to make in our work is that whatever we do will always be misused in the worst possible way. If we explore that idea, it might give us some sense for how to structure our output to curtail the worst of the damages.
> The early days of search in Google or social in Facebook probably didn't elicit the same kind thought process as a lethality engineering post.
It did for at least one person (me). I was 16 in 2004 with 11 years of dev experience, trying to decide whether to go out to SV, go to college for CS, or do something else. I was from the same city/community as Larry Page and in Zuck's age group, so it wasn't an absurd consideration to try. Lots of things went into my decision to do something non-CS related for college, but morals were one of the reasons I didn't go to SV (I objected to the professionalization of the web + Zuck creeped me out + I didn't agree with cutting out humans/curators from the search process like Google did).
It's just that until very recently, people either thought I was lying OR that I was just batshit insane. Who is invited to a gold rush and doesn't go?
I think you were hard on him. There should be no ethical qualms when our weapons are used on enemies who seek to kill us or attack our interests.
Also, if an ethical person doesn’t take this job, someone far more unethical probably will. And they will raise no objections if they should ever be necessary. Kind of like how a lot of bad people become police officers when no one good wants to do it.
I can relate to that. In my career I stepped out and into defense, and it never really bothered me that much to be honest. But then it was always things like fighter jets and helicopters sold to NATO members, I never had to rationalize that we build the weapons carrier and not the, e.g., missiles that actually cause harm.
I always drew the line at small arms so. Way more people die because of those, they end up in every conflict and there have been too many scandals of those smalls arms manufacturers circumventing export restrictions. Quite recently I added supporting countries like Saudi and the UAE to that list, even the job would have been really interesting, providing highly sophisticated training services to the Saudis is nothing I could do and still look myself in mirror. And civil aerospace is fun as well.
I worked in defense too, might go back. When I get calls I'm like "I don't do work for the Saudi's or the DEA" and half the time the recruiter is like "Uh, I said this job is for Raytheon."
A friend is a very good university lecturer in physics, and a pacifist. He isn't particularly please about the fact that a decent number of his students will turn the particular lessons he teaches towards the production of weapons.
Lethality Engineer? Is that a P.Eng. kind of position? If your work doesn't actually kill anybody, could you be sued for malpractice and lose your license?
So, the gist of this article is "oh look, machine learning can be used to build super-weapons!". Of course it can, and we have no shortage of (other) tech to make our life miserable. And of course the problem is not the tech itself but the people and their institutions. That paper from Urbina et. al. is at best academic click-bait. They are optimizing for publishing something and get cited, not for social good. They should stop.
Interesting exercise, perhaps the harmful molecule generating AI still generates helpful molecules because molecules harmful at a certain dose may sometimes be very beneficial in a (much) lower dose. And the other way around of course.
Perhaps we should simply have one “biologically active molecule” generating network. The dose will ultimately determine the toxicity.
So, as I said my remark didn’t come out right, but, some molecules may be considered harmful at low dose and harmless at high dose if they stabilize a deteriorating conduction at or over some threshold concentration. Yeah I know it’s a fetch but you got me thinking… It’s not that clear cut.
I mean the urine of someone on chemotherapy is pretty toxic, still we consider the molecules beneficial to the patient overall (the patient-tumor system if you will, not the patient by themselves).
Software is able to simulate the effect of chemical compounds / molecules on the human body. This can be used to find drugs that do specific things, or stronger versions of existing drugs. For example, you could look for very strong but very short acting sleeping pills that immediately make you fall asleep, but cause zero grogginess the next day. Or you could optimize antibiotics to have a high half life, so you only have to take them once, instead of 3 times a day for a week, which you can easily forget.
Now think about nerve gas. We have discovered lots of different nerve gas agents and know pretty well how much of each type you need to kill a human. Said software can be used to find new versions of nerve gas that kill with even lesser concentrations. You could also optimize for other variables: Nerve gas that remains on surfaces and doesn't decay by itself for example.
They had an AI that looked for safe drugs by minimizing an estimate of lethality, changed it to ‘maximize’ and the computer spewed known nerve gas agents.
Why make a chemical weapon when you can just tweak a virus which self-replicates?
BA.2 is even more infectious than BA.1 which is saying something, imagine an engineered BA.3 with even more spread and then make it as even more deadly. You might even be able to target it to one race or region if there is a gene specific to that area.
Always hoped the future would be Star-Trek-like but it seems all it takes is one dictator or terrorist to end the world, slowly at first but then it would double every other day and impossible to stop.
So, their tool will draw molecules that are good at doing harm, and that is it? No word on stabilization (which makes it safe to handle), synthesis, purification and such. I'd wager that most of these substances have at some point been on somebody's blackboard, but deemed impractical or infeasible, and then not pursued, and that's why we don't know them by name today.
This is not really that worrying IMO - we already have weaponized toxins, viruses, and enough explosives to blow up the entire planet. So what if an AI can come up with something a little bit worse? It isn't the existence of these things that's stopping us all from killing each other.
The use of nuclear weapons would be... obvious: if an explosion in the 10Ktn or bigger happens, it's a nuclear weapon. There aren't enough nuclear powers to make the use of nuclear weapons plausibly deniable.
The use of chemical weapons might not be as obvious if they are slow acting. And the production of chemical weapons is much easier than that of nuclear weapons. Though, the dispersion of chemical weapons is non-trivial.
The use of biological weapons need not be obvious at all -- "it's a naturally-evolved pathogen, this happens!". The development and production of biological weapons is much easier than that of nuclear weapons. Human and animal bodies can be made to help spread biological weapons, so their dispersion can be trivial. The only thing that a bioweapons user might need ahead of time is treatment / vaccines, unless the bioweapon is weak and the real weapon is psychological.
Is anybody searching for compounds that reduce evil intent? Something that would mellow people out without causing hallucinations. A mass tranquilizer? Not effective against lone operatives but able to be deployed against an invading army.
On a more serious note, anything that's going to affect behavior is going to have a dosage range. Too little absorbed, and there won't be enough effect. Too much, and that will harm / kill people in interesting ways.
With chemical weapons, you only worry about the bio-accumulating enough to kill your enemies. An enemy receiving more than a lethal dose isn't a problem.
> The M44s relatively small production numbers were due, like all U.S. BZ munitions, to a number of shortcomings. The M44 dispensed its agent in a cloud of white, particulate smoke.[3] This was especially problematic because the white smoke was easily visible and BZ exposure was simple to prevent; a few layers of cloth over the mouth and nose are sufficient.[5] There were a number of other factors that made BZ weapons unattractive to military planners.[5] BZ had a delayed and variable rate-of-action, as well as a less than ideal "envelope-of-action".[5] In addition, BZ casualties exhibited bizarre behavior, 50 to 80 percent had to be restrained to prevent self-injury during recovery.[5] Others exhibited distinct symptoms of paranoia and mania.[5]
Of course they are. Among the "evil intent" they would reduce is any desire to rebel against your government, so you bet all big intelligence agencies are looking into it, for instance. Science fiction wrote about this decades ago.
Fortunately, there's a lot of considerations involved in deployment of anything. It's easier said than done to get something of a medical nature into a population surreptitiously, because it's hard to get a certain dose into one person without someone else getting not enough and yet someone else getting way too much. You'd have to come up with a way of delivering a medical dose in a controlled fashion and lie about it or something, you couldn't just sneak it into the food/water reliably.
Further, just because someone can name the exact complicated effect they'd like doesn't mean there's a drug that corresponds to it. Serenity, already mentioned, is a bit of silly example in my opinion because such a large effect should have been found during testing. But it does no good to pacify the population such that they'd never dream of so much as peacefully voting out the current leaders if the end result is that nobody would ever dream of so much as having enough ambition to show up to their jobs and you end up conquered by the next country over without them even trying, simply because they economically run circles around you. Or any number of other possible second-order effects. In a nutshell, it's dangerous to try to undercut evolution just to stay in power if not everywhere decides to do so equally, because you'll be evolved right out along with the society you putative rule. Evolution is alive and well and anyone who thinks it's asleep and they can screw around without consequences is liable to get a lethal wakeup call.
> The report states: “These findings, which are consistent with the finding in clinical trials that lithium reduces suicide and related behaviours in people with a mood disorder, suggest that naturally occurring lithium in drinking water may have the potential to reduce the risk of suicide and may possibly help in mood stabilisation, particularly in populations with relatively high suicide rates and geographical areas with a greater range of lithium concentration in the drinking water.”
The "dual use" paper this is commenting on is the clickbait equivalent of "encryption is for pedos", and maybe Derek's "not too surprising" is code for "Science editors are not discerning enough".
Like, this is the whole point of pharmacology: Predicting the biological interactions of chemicals (what they do to biological targets, how potent), and their ancillary physical properties (solubility, volatility, stability etc). For example,
Optimizing for mu-opioid agonist activity gives you super potent painkillers, drugs of abuse, and that stuff Russia gassed a theater with to knock out / kill hostages and kidnappers (i.e. fentanyl analogues)
Optimizing for inhibition of various proteases might give you chemotherapy drugs with nasty side effects, or stuff with nasty side effects and no known therapeutic use (i.e. ricin)
Optimizing for acetylcholinesterase inhibitor activity will turn up nasty poisons which could be purposed as "nerve agents" or "pesticides"
Optimizing for 5HT2a activity will give compounds that are great for mapping receptor locations in brains, which are also drugs of abuse, and which are also lethal to humans in small doses.
And the "predicted compounds not included in the training set" thing is just table stakes for any predictive model!
> 5HT2a
You sure you don’t mean 5HT2b?
I mean, anything can be toxic with enough dose, but the b-subtype agonists seem a lot more toxic than the a-subtype agonists.
(Fun fact: 6-APB, a “research chemical” recreational substance became an actual research chemical because of it had better 5HT2b selectivity than what was previously used in lab research)
Was thinking of halogenated NBOMe series - observed in humans to have a pretty narrow therapeutic index re: death, cheapish synth, can be vaporized
But yeah 2b could be worse. Or many other targets as well
Funny thing, optimizing "research chemicals" for (1) uncontrolled synthetic pathway and (2) potency is common to Institutional, Druggie, and Terrorist researchers. None of them want to go through the bureaucracy for controlled substances and potency is good for [better controlled experiments / smaller quantities to transport / more killing power]
1 reply →
Fortunately we don't see any real work in the chemical and biological weapons space anymore. While it would still be pretty handy for terrorist groups, in actual warfare chemical weapons aren't super useful. See https://acoup.blog/2020/03/20/collections-why-dont-we-use-ch... .
On a tangent: it occurred to me recently that we also don't see much use of ICBMs with non-nuclear payloads, despite these being a fairly-obvious "dominant strategy" for warfare — and one that isn't banned by any global treaties.
I'm guessing the problem with these is that, in practice, a country can't use any weapons system that could potentially be used to "safely" deliver a nuclear payload (i.e. to deliver one far-enough away that the attacking country would not, itself, be affected by the fallout) without other countries' anti-nuke defenses activating. After all, you could always say you're shooting ICBMs full of regular explosive payloads, but then slip a nuke in. There is no honor in realpolitik.
So, because of this game-theoretic equilibrium, any use of the stratosphere for ballistic weapons delivery is effectively forbidden — even though nobody's explicitly asking for it to be.
It's interesting to consider how much scarier war could be right now, if we hadn't invented nuclear weapons... random missiles just dropping down from the sky for precision strikes, in countries whose borders have never even been penetrated.
> despite these being a fairly-obvious "dominant strategy" for warfare
I don't think these are quite as viable as you think. ICBMs are expensive. Probably tens of millions of dollars each, for a single-use item. Cruise missiles cost $1-$2 million to deliver the same payload and have a better chance of surprising the enemy.
ICBMs have longer range, but how often do you need to strike targets more than 1000km past the front line? They're inherently strategic weapons.
31 replies →
Conventional Prompt Global Strike is intended to provide the ability to deliver a conventional kinetic attack anywhere in the world within an hour. It has been an active area of weapons research for the US for 20 years. As you speculate, misinterpretation of the launch is a concern. [1]
As opensocket points out, there are many shorter range conventional weapons used across borders. The cruise missiles of gulf war 1 or the drones of the post September 11 world.
[1] https://sgp.fas.org/crs/nuke/R41464.pdf
To tie this in with current events this is exactly what makes the no-fly zone idea in Ukraine so dangerous. All of the things you have to do to establish a no-fly zone and take away the enemy's ability to fire into and effect your no-fly zone look the same as a prelude to an invasion or nuclear first strike. This is made worse by the fact that many of the weapons systems you would be using are dual use. Meaning they were designed to deliver conventional or nuclear weapons. It's a massive gamble that the actions won't be misinterpreted or used to justify moving up the escalation ladder.
There are quite a lot of shorter-range conventional systems. In practice you don't need that inter-continental range for most purposes. For some modern examples you have the Chinese DF-21 and the Russian Iskander system. And a lot of those systems are dual-use: capable of delivering both nuclear and conventional payloads. It's not totally clear what that will mean in a conflict between two nuclear powers. What do you do when early warning radar picks up a ballistic missile coming in when you can't tell if it is nuclear or conventional? Plus this isn't a video game, you won't hear some alarm going off after it detonates indicating it was a nuclear explosion. You'll need to send someone to do a damage assessment, and that takes time.
4 replies →
> It's interesting to consider how much scarier war could be right now, if we hadn't invented nuclear weapons... random missiles just dropping down from the sky for precision strikes, in countries whose borders have never even been penetrated.
Why are cruise missiles any less scary? They are indeed used in precision strikes across country borders, and can kill you just the same. The existence of nuclear weapons still allows some countries to use cruise missiles, as we see happen almost every year.
I'd consider how much more docile the nation's of the world would have become sans nuke.
If the possibility of an untraceable, space borne, hypersonic weapon was on the table we might have had a better deterrent than nuclear weapons. The lack of fallout and total deniability makes it almost certain they would have been deployed and quite concisely ended a few conflicts at the onset.
It is alarmingly frightening, moreso even, because the impact could be extremely precise- leaving infrastructure intact.
28 replies →
Shorter range ballistic missiles have been heavily used in multiple conflicts around the world for many years. The US military has been researching the possibility of using conventionally armed long range ballistic missiles to fulfill the prompt global strike mission. Potential target countries have no anti-nuke defenses. But there is a risk that Russia or China could misinterpret a launch as aimed at them.
in WW2, Germany used V2 missiles for indiscriminate bombing of cities (primarily London). I can imagine it would look like that, but worse - and having gone to a few museums that showed Blitzkrieg London, that was bad enough as it is.
3 replies →
Interesting tangent I hadn’t considered before. However, China is testing some ballistic anti ship missiles. https://www.navalnews.com/naval-news/2021/11/aircraft-carrie...
To be fair, if these become a reality, they would likely strike targets in the Pacific Ocean and South China Seas, far away from the US, but the potential to spook nuclear nations is still there.
i would imagine delivering a conventional warhead with an ICBM has a very high risk of being mistaken for a nuclear armed ICBM. Also, they're expensive. Putting a JDAM package on an old iron bomb turning it into the most advanced precision guided munition is very cost effective.
https://en.wikipedia.org/wiki/Joint_Direct_Attack_Munition
>game-theoretic equilibrium
Equilibriums change, every major US platform was at one point designed to be nuclear capable, i.e. cruise missiles now liberally launched from planes/bombers/ships that are all nuclear capable. There's no a reason nuclear countries who get attacked by any US platforms should assume any incoming ordinance ISN'T nuclear, down to gravity bombs, except for expectation - knowing US has overwhelming conventional capabilities and would rather use it than nukes.
Same will apply as conventional ICBM matures - we haven't seen much of it because ICBMs have not been sufficiently accurate unless carrying nukes where CEP in meters don't matter. For countries with power projection, it was dramatically cheaper to get closer first and deliver less expensive ordinance. Conventional ICBMs seem effectively forbidden because most actors assume they're too inaccurate for anything but nukes and too expensive for anything but nukes.
But that's changing - there are hints that PRC is pursuing rapid global strike i.e. US prompt global strike, because IMO it's the great equalizer in terms of conventional mutually assured destruction precisely because it isn't banned. A lot of articles being seeded on SCMP about PRC hypersonic developments that spells out meter level CEP ICBMs designed to conventionally attack strategic target of depth, aka Prompt Global Strike.
Ergo (IMO) PRC maintaining no first use nuclear policy while conducting massive nuclear build up to setup credible MAD deterrence. This sets up the game theory of accepting that conventional ICBM attacks on homeland from across the globe is possible and that it's best to wait for confirmation unless one wants to trigger nuclear MAD. Entire reason US / USSR and countries that could moved to Nuke triad or survivable nuke subs was because it bought more time than hair trigger / launch on warning posture.
This makes a lot of sense for PRC who doesn't have the carriers, strategic bombers or basing to hit CONUS (or much outside of 2nd island chain). It makes a lot of sense for any nation with enough resources for a ICBM rocket force but not enough for global force projection (basically everyone). World will be very different place if such capabilities proliferate. Imagine any medium size country with ability to hit stationary targets worldwide - fabs, server farms, power stations, carriers being repaired in a drydock.
2 replies →
I’m not sure there’s much difference between the Russians using air-launched cruise missiles (with ranges of hundreds to potentially thousands of kilometers and almost always capable of carrying a nuclear warhead) launched from their Tu-95 Bear strategic bombers (equivalent to the B-52), which Russia has done several times now in Ukraine.
1 reply →
> So, because of this game-theoretic equilibrium, any use of the stratosphere for ballistic weapons delivery is effectively forbidden — even though nobody's explicitly asking for it to be.
Interesting! SpaceX was hoping to one day use Starship for quick intercontinental flights. I wonder if this unspoken rule would make that prohibitive?
5 replies →
There could also be effective anti-missile defenses, which are currently politically impossible due to the fact that developing them puts you outside of the effectiveness of deterrence.
take a look at https://en.m.wikipedia.org/wiki/Prompt_Global_Strike
Chemical and biological weapons are very useful in warfare as a way to demonize one combatant. False or doubtful claims of chemical weapons deployments have an effect on the response of the public and international organizations that is entirely out of scale with the damage that could be inflicted.
Relevant if you want real life cases of military impersonation: https://en.wikipedia.org/wiki/False_flag
2 replies →
Just doesn't seem like it's worth it even for terrorists.
Why invest a bunch of time and effort making more and more deadly poisons when we've already got a wide variety of them that are cheap to manufacture, well known in how they work, and don't cost a bunch of research money to uncover?
At the risk of putting myself on a watchlist, They already know that, so They have an eye on certain kinds of labware, different precursors, and such. And They already have antidotes to some of these poisons.
One could optimize for compounds with hard-to-monitor precursors. Compounds that can be transported with low vapor pressure and volatility, so they cannot be easily sniffed out.
Or imagine a lethal compound with a high delay factor. Or something with specifically panic-inducing effects, perhaps hemorrhagic with a side effect of your skin sliding off in great slick sheets. Another interesting high delay factor compound might induce psychosis: have fun tracking where these gibbering maniacs were a week ago.
With a sufficiently dark imagination, "needs" could be identified for all sorts of compounds.
Remember, the goal is to throw a monkey wrench into the gearwork of an opposing civilization, not necessarily to kill. Fear of the unknown is very effective for this.
36 replies →
True, though you have to remember that threat is a social construct and isn't necessarily a rational measure. The 2001 anthrax attacks killed 5 people, injuring 17, and shocked the nation. As a direct result Congress put billions into funding for new vaccines and drugs and bio-terrorism preparedness. If 5 people were killed and 17 wounded in a mass shooting by a terrorist, would we really have reacted as strongly?
If you wanted to install fear into a country, I think being attacked by some custom, previously unknown chemical weapon would scarier than sarin.
from what i understand the delivery of a chemical or biological weapon is the hard part. For most things, you can't just pour it out on the ground to have a huge effect. Somethings you certainly can, weapons grade Anthrax probably just needs a light breeze to devastate a city but something like that is beyond the reach of your average terrorist groups.
Hmmm, and then you have to ask, for whom is it worth it?
Who might invent a bunch of time and effort in those areas?
I guess developments are not published on Nature.
They are amazingly useful in real warfare. Drop nerve gas on a city, walk in a couple days later. WMDs are the only way to really take a country by force, and of all of them, chemical weapons are the most palatable and also the easiest to produce.
Considering this, defense against them is at least mildly important. A proper defense only exists by considering offense, so they're still developing chemical weapons somewhere. The modern hot topic is viruses and other pathogens.
That's not how it works, and they're not very useful at all. The amount of actual product you need is non-trivial and at that point you might as well just use modern conventional munitions.
The reason why it fell out of favour isn't because it's dangerous, it's because it was ineffective outside of TV and film.
4 replies →
They go against the Geneva Protocol, and it's not even allowed to stockpile them so not even useful if you are a terrorist with a death wish because then there are simpler ways to end your problems.
8 replies →
> Fortunately we don't see any real work
Not really, having new developments all classified is not helpful to anyone.
The real fun starts when somebody starts using techniques like this that overcome the weaknesses of known chemical weapons and provide specific advantages. It's also kind of hard to monitor computational chemical research.
It's my understanding that the Soviet army doctrine in the '70s and '80s included the use of chemical weapons. That hypothetical threat put a hell of a lot of friction on NATO in terms of training, supplies, and preparedness.
that we don't see it doesn't mean it is not happening.
This will probably come in handy for industrial espionage type tasks.
Lets say you had a nation-state enemy who eats a lot of some ethnic ingredient. Come up with a cheap artificial flavor/color or process that is optimize to give heavy consumers cancer in 30 years. Not in one year, that will show up in the approval process. Then have an agent in the target country "discover" thru random chance this really excellent food dye or whatever.
Now you kill half the population with cancer, you're gonna get nuked in response, even non-nuke countries will be pissed off enough to get nukes just to nuke the perpetrator. But lets say you make the victims fat and sick and die a little younger just enough to get 1% hit on economic growth...
Some people would say this is how we ended up with trans-fats and margarine and vegetable oils in general or certain veg oils in specific.
Certainly, corn syrup has caused more human and economic devastation that fission, nerve agents, or most any WMD I can think of...
The problem with this is that this is basically genetic engineering, you might successfully make a low-level economic growth impact now, but future generations will be resistant to the poisons as those weak against them die off. You are securing your own demise long-term if you don't subject your population to the same.
This is far more fun than believing in an emerging accident. You don't have to eat corn syrup, btw.
I guess if it's tasty then it's fair game.
If this were really a practical concern, machine learning would be designing drugs that fly through the clinic today. They aren't and so this paper, though click-grabbing, is probably of no practical consequence.
One reason is lack of data. Chemical data sets are extremely difficult to collect and as such tend to be siloed on creation. Synthesis of the target compounds and testing using uniform, validated protocols are non-trivial activities. They can only be undertaken by deep pockets. Those deep pockets are interested in return on investment. So, into the silo it goes. This might not always be the case, though.
For now, the paper does raise the question of the goals and ethics around machine learning research. But unintended and/or malevolent consequences of new discoveries have been a problem for a long time. Just ask Shelley.
A successful drug candidate must be useful in the treatment of human medical problems and not have harmful side effects that outweigh its benefits. A weaponized poison may have any number of harmful effects without diminishing its utility. A compound with really indiscriminate biochemical effects, like fluoroethyl fluoroacetate, makes a potent poison without any specific tuning for humans. It's much easier to discover compounds that genuinely harm people than those that genuinely help them.
"Now, keep in mind that we can't deliberately design our way to drugs so easily, so we won't be able to design horrible compounds in one shot, either. "
I would discount this, heavily and concerningly, as a false sense of security. The reality is that prohibitive factors in creating new drugs from compounds discovered similarly (by AI or other automated process) is almost entirely due to testing safety procedures and regulations... If the bad actors are trying to find the most lethal compound with no such oversight - and chances are very high that they aren't bound by any such regulation if they're state-level labs operating under impunity - there is nothing but the synthesis that would make the formulation and testing of these as impractical as the author claims. Take away the years-long, heavily scrutinized and regulated multi-stage billion-dollar path to drug approvals and you'll find that barrier is not so high.
I would like to think this data could be helpful to any organizations looking to proactively develop detectors or antidotes for such compounds - especially if the threat was previously unknown to them.
Let's say an entirely novel class of toxin was found in a cluster of these predictions that has no existing references in private or public records - it could be that another organization has discovered and synthesized something similar through one of many other paths.
Many lines are drawn between this type of approach and that of whitehat hackers. You must necessarily create the vulnerability to mitigate it. It feels like "white hat" biolabs claiming the same are operating on the same conundrum and that the difference between "studying for the sake of mitigating" and "creating a weapon" are fundamentally indistinguishable without an absolute knowledge of intent - such is impossible from the outside.
> The reality is that prohibitive factors in creating new drugs from compounds discovered similarly (by AI or other automated process) is almost entirely due to testing safety procedures and regulations
Most drug candidates fail because they don't work, not because of any regulatory procedure. About 50% of drug candidates that enter Phase III trials--the final clinical trial before approval--fail, and that's almost always because they failed to meet clinical endpoints (i.e., they don't do what they're supposed to do), and not because they're not safe (toxicity is Phase I trials).
That "not working" part has some nuance to it as well. How well do we predict ADME? Is there binding with some off target protein that makes it terrible? Maybe it just doesn't bind to the desired target at all.
Toxins don't have those constraints, its not even about regulation. Making something that's safe is way harder than making something that is not safe, purely because of the complexity involved in making the thing safe.
I was thinking this as well. If a new drug works well for 99% of the people, has mildish side effects for 0.9% and is really bad for 0.1% of people, that's no good. But if a nerve agent kills 99% of people and is not effective on 1%, that's just fine.
Also, compounds that are fatal tend to be so to all life forms vs just humans with the variation being dosage. It goes without saying the odds of taking a compound and finding a drug that does one very specific thing without doing anything else.. and demonstrating that it's safe in people, is orders of magnitude less likely than finding a compound that is lethal.
Hype over crap like this grinds my gears. Organophosphines like VX are ALL toxic. There's about a zillion such toxic molecules all containing the same functional group. This study does not demonstrate that this tool is better generator of toxic molecules than anything that includes the basic rules of valence and rudimentary understanding of shape similarity.
When thinking about whether ML does something novel, we must always compare with some simple alternative. I would be impressed if it'd predicted something like Palytoxin, a highly specific molecule with extraordinary toxic activity. There's no way the tools of this paper would though.
-- director of ML at a drug company.
> ricin is (fortunately) not all that easy to turn into a weapon, and the people who try to do it are generally somewhat disconnected from reality (and also from technical proficiency).
Of course that fact was no barrier to much hype about the "dangers" it posed, either. I suspect the same now; that we have more to fear from the fear junkie propaganda than the actual facts.
I personally fear the lone-wolf attack drastically reducing in cost and effort. Where it would once be cost-prohibitive to design and manufacture your own nerve gas or lethal virus, these days with AI/ML and Crispr-cas and the like, it feels like any intelligent, deranged person wanting to take as many people to the grave with him has the tools to do just that.
Intelligent and deranged persons already have the tools to make way more casualties with way less effort using guns and/or explosives. The "problem" for them is that people who get sufficiently deranged to think that killing a few hundred (or even thousand) people will meaningfully solve the problem they are upset about will also be sufficiently deranged that their ability to reason coherently will be drastically reduced.
6 replies →
I think this is inevitable and something we will grapple with in coming decades. Especially around genetic engineering of viruses.
>>The thought had never previously struck us. We were vaguely aware of security concerns around work with pathogens or toxic chemicals, ...We have spent decades using computers and AI to improve human health—not to degrade it. We were naive in thinking about the potential misuse of our trade...
Of course now, the next step is to use the technology to preemptively search for and develop antidotes to the new potential weapons their tool has discovered.
We already (sort of) do this. AI/ML is probably used for simulating nuclear explosions, and is [arguably] even more useful and accurate than actually setting off a bomb, and measuring it.
It makes sense that it could be weaponized. When Skynet becomes self-aware, it would probably just design a chemical that kills us all, and would aerosol that into the atmosphere. No need for terminators, just big pesticide cans.
I don't think AI/ML is really used for simulating nuclear explosions. There's not much point, better techniques exist.
What such better techniques exist?
3 replies →
I'm quite sure we have already invented several chemicals that match your description -- like sarin gas, invented in 1938 by someone who, indeed, wanted to create a decent pesticide. A lethal dose of sarin gas is something like 28-35m³/min over 2 minutes exposure, according to wikipedia. [0]
Hitler was well aware of its creation, and I believe quite a lot of the stuff was produced for the purpose of warfare. There were several in the Nazi military who wanted to use it, but Hitler declined.
That seems rather odd, given his indifference to exterminating people with gas on an industrial scale beyond the theater of war. It has been suggested that Hitler was probably aware that to use sarin gas would be to invite the allies to do so in response, which would result in a dramatic loss of life on the German side due to the sheer lethality of such chemical weapons. [1]
Perhaps he thought it easier to stick to conventional warfare, in which the pace is more manageable than with WMDs, where you would start going down the road of mutually assured destruction but without the strategic framework in place to prevent anyone from actually wiping out a population before realising how bad an idea it would be.
And I think this reluctance to change the game, this seemingly deliberate moderation, perhaps best demonstrates the true difference between the machine and human in warfare.
It is not a difference in innovation -- we have always been very good at inventing highly optimised ways to end life.
The difference is that a machine intelligence will not hesitate. It will not ask for confirmation, pause or break a sweat. It will pull the trigger first, it will point the bombs at anything that is an adversary and anything that could theoretically be or become an adversary, and it will not miss. And it will not have to face ethical criticism and historical condemnation afterwards. [2]
[0]: https://en.m.wikipedia.org/wiki/Sarin
[1]: https://www.washingtonpost.com/news/retropolis/wp/2017/04/11...
[2]: Assuming this is a Skynet-like machine intelligence, which doesn't really have the capacity for remorse or negotiation and seems primarily, indeed solely occupied with the task of ending human life.
Obviously, a true AI that is essentially a conscious mind equivalent to our own minds, may experience the same hesitancy that most of us would, were our fingers to be over the buzzer.
Unless the AI independently arrives at a different set of values to us, like the Borg or something.
Chemical weapons are expensive. Consider the logistics and training required to effectively deploy them, plus any specialized equipment. Meanwhile, they're only useful as long as your opponent doesn't know you're planning to use chemical weapons, since countermeasures are relatively cheap and every major military knew what to do about them by the time WWII broke out. As soon as your enemy knows to beware chemical attacks, all you're doing is annoying them while making it hard for your own troops to advance (they have to put on chemical suits/masks themselves, or else wait for the gas to disperse). Very hard to use effectively in maneuver warfare. They didn't even prove very effective in WWI, which was much closer to an ideal environment for their use.
Its the classic logistics problem. Scaling ratio of weight vs volume or something like that. Just like nukes, if you heat an enemy soldier to 100M degrees he isn't any more dead than heating him to 10M degrees and volumes expand very slowly with mass so making bigger and bigger bombs is a fools-errand.
Same problem with chem weapons. You hit a tank brigade with 1000x lethal dose they aren't any deader than if you hit them with 1x dose. But if the bomb misses which is likely, all you've done is REALLY piss them off. Nerve gas in an empty wheat field just kills a bunch of corn bugs but it really pisses people off. If you target their tank brigade and miss, they'll target your home town, as we did to them with conventional bomb even without having been nerve gassed to start with. If you target their home town then the brigade you missed is going to be unhurt and really angry. Its the kind of weapon thats pretty useless unless you have infinite supply and infinite logistics. Like cold war USA or cold war Russia.
The allies had better logistics than the Germans so they knew the second time around in WWII that trying to go chem is just going to end up in the German's getting more chem'd than the allies.
Another issue is WWI and previous its all about siege warfare and breaking sieges where WMD is awesome and useful, whereas WWII and newer is all about maneuver warfare and blitzkrieg and all of Germany's plans and all of their early success were based on the idea that anything in range of shells or aircraft today is going to be occupied rear supply area next week at the latest, so destroying it would be pretty dumb because we need that area to be the rear of the battle space next week. For a modern comparison the USA could have nuked the green zone in Iraq and there's absolutely nothing anyone could have done about it, but 'we' knew we'd be occupying the green zone and needing something like the green zone, and the green zone is sitting there for the taking, so in an incredibly short term perspective it would have saved troops and saved time and saved effort to just nuke it instead of taking it the old fashioned way, but in medium and longer term it would be counterproductive to war efforts to use WMDs against the green zone, so we didn't.
Hitler probably also experienced gas (not sure his generals did, though). People forget that he was actually a decorated NCO, from WWI (which had a lot to do with his terrible attitude, later in life).
It was fairly worthless, militarily. High risk, big mess, no real tactical advantage, and it just pissed everyone off. Its only real efficacy would have been for bombing civilian targets, and I don't think they had the delivery mechanisms.
[dead]
The article assumes that full development of a new chemical weapon would require more development effort. With regards to military usage: storable at room temperature, relatively easy to manufacture from commonly available precursor chemicals, etc. [1]
How true is that? Are there components of this process that make things easier now? Where I have chemical structure X, and a system generates the process steps and chemicals needed to produce X. How much of the domain in chemistry / chemical engineering has been automated these days? What are the future prospects for this?
[1] I assume one of the design goals for a new chemical weapon for military use is that it breaks down in the environment, but not too quickly (like say in a week or a month). Though I suppose if you want to just destroy civilization you would design for longevity in the environment instead. And being able to seep through many kinds of plastic if possible.
> Where I have chemical structure X, and a system generates the process steps and chemicals needed to produce X.
Undergraduate chemistry students spend a fair amount of time learning how to look at a novel structure X and by disconnecting "backwards" it into simpler components, deduce a route by which it might be synthesed "forward" in the laboratory from readily available starting materials.
There's an excellent book on this, "Organic Synthesis: The Disconnection Approach", by Stuart Warren.
Interesting.
Were / are you a chem major?
Any other major topics or readings you could recommend for someone wanting a general understanding of key concepts in modern chemistry? I'd suppose generally: materials, synthesis, o-chem, and chem-eng.
My own background: began a hard-science degree. One year undergrad uni chem.
2 replies →
While it's worrying and worth thinking about, the track record of using AI to generate pharmaceuticals to do good has been "mixed", except really it's just been a bust. It may someday do great things, but not much yet, and one silver lining is that AI-generated toxins are unlikely to improve on the human-designed ones, either.
"That is, I'm not sure that anyone needs to deploy a new compound in order to wreak havoc - they can save themselves a lot of trouble by just making Sarin or VX, God help us."
> the track record of using AI to generate pharmaceuticals to do good has been "mixed", except really it's just been a bust.
Researchers have only been using AI for drug development for like 6 years, I think it's way to early to call it a bust
I guess I should have said "...thus far".
Wow - I picked up on this earlier today, and even quoted the same as in this article. I was amazed that the scientists had not considered that AI could be/is being used for harm. (Was downvoted for this, but whatevs.)
It struck me as incredibly naive, but then - what would someone else do in their situation? Most of us work in silos without awareness of how our work is used, and I suspect we are often causing (unintentional) harm to others whether we are scientists, programmers, in finance, in health, in government, etc. If we realise our predicament, there isn't an moral authority to make things right. There is only the legislation that was been written by lobbyists paid by the corporations we work for.
Putting the article in broader context, perhaps it is about the creation of a moral framework for AI intended to pacify our disgust at the system we find. I expect that we will be expected to look away as AI "ethics" committees justify the unjustifiable, but call it ethical. As whatever-it-is is found to be ethical after all by ethical authorities, most of us we will wave this through and consider that we have acted judiciously. IMO.
Chemical weapons above a certain level, bio weapons and nucs are all seen as weapons of mass destruction and are not really that useful tactically. Introducing strategic-destabilizing elements to a conflict greatly increases its unpredictability and probably is a health risk for the leaders involved.
This is essentially what the pesticide and herbicide industries have been doing since their inception, i.e. designing molecules that efficiently kill animals, insects and plants. It seemed like a miracle at first, but the long-term consequences of things like persistent chlorinated aromatics and their derivatives (Agent Orange and dioxin for example) eventually appeared in human populations.
The development of the toxic nerve agents (organo-phosphate compouds mostly) in particular was a side effect of research into insect toxins. The nerve agents were discovered in this manner, they worked too well. Nevertheless, these pesticides were deemed safer than the organochlorines because they degraded fairly rapidly after application (although they are implicated in nerve damage related diseases like Parkinson's in agricultural areas).
Insect infestations are indeed a big issue in agriculture and can wipe out entire crops if not dealt with, but there are plenty of options that don't require applications of highly toxic or persistent chemicals.
Otherwise, this is just another of the many issues modern technology has created. Smallpox is another one - in the late 1990s, there was a great debate over whether to destroy the last smallpox samples - and then in the mid 2000's, someone demonstrated you could recreate smallpox by ordering the appropriate DNA sequences online and assembling them in a host cell. Then there's the past ten years of CRISPR and gain-of-function research with pathogenic viruses, a very contentious topic indeed, and still unresolved.
Many years ago a colleague who works in defence told me about a job posting he'd seen but was having a moral struggle with.
The opening was for "Lethality Engineer": Ideal candidate with good physics and medical background.
I said that the main perk was that at least on Halloween he wouldn't need to buy a costume. He could just go out as himself.
He didn't take the job.
I hope recent events have illustrated that if it weren't for the people who develop lethal weapons, we (as in you and I) would be helpless against the bullies of the world. Unilateral pacifism is cute philosophy only when there are rough men standing ready to do violence on their behalf.
I strongly agree with this sentiment. However, it is hard for an ethical person to participate in developing war technology when possession and usage of the weapons is purely a political question, and history also has seen our side of geopolitics commit atrocities.
My stance has previously been that I am unwilling to work on weapons technology, because history has shown that these weapons sometimes end up being used for an indefensible cause. Then all of a sudden you're an accomplice to murder, and getting away with it.
In the light of Russia's invasion of Ukraine, which is just a continuation of its historical imperialism, working on weapons is something I would be perfectly okay with and probably even motivated to do. But stop for a moment and think what a history of aggressive military actions does to our society's ability to recruit for this important job.
13 replies →
The flaw in that logic is that, if it weren't for the people who develop lethal weapons for the bullies, we wouldn't have to fear the bullies.
Also, I think the design space of "radical defense" is under explored. Our (western) armies are still designed for attack and force projection, although we have long since renamed our war ministers secretaries of defense.
But I wonder if you could develop defense capability to make your country unattackable. Not by threat of retaliation, but for example by much much stronger missile defense. Or by educating ("indoctrinating") your own population, so that an occupier would not find a single collaborator? Or by mining your own infrastructure, and giving every citizen basic combat training (a bit like the swiss)? Or by fostering a world-wide political transformation that is designed to prevent wars from happening at all?
I think if we wanted to spend money researching stuff to keep us safe, it doesn't necessarily have to be offensive weapons.
4 replies →
Especially since we cannot really help Ukraine with anything but tech and they are outnumbered, so the only advantage we can give is how much better our weapons are than Russian ones.
Have you considered, though, that we (as in you and I) might be some of these bullies in the world and that these rough men aren't just standing by, but are actively doing violence on our behalf ? One needn't look much further past recent events to find examples aplenty.
I understand the point you are trying to make, but it's not as easy as pretending that the weapons "we" develop are purely for morally and ethically righteous purpose.
With respect you may be making some unfounded assumptions about what I've built and what I believe.
My point was really about the fact that this job title "Lethality Engineer" actually exists. And moreover, that it asked for medical qualifications, which would go against any doctor's Hippocratic oath.
Most of us who've done defence related work are happy at the edges, with tactical information systems, coms or guidance (my stuff ended up in targeting).
But when it comes down to figuring out how fragments can be arranged around a charge to make sure the waveshape optimally penetrates as many nearby skulls as possible... hmmm suddenly not so gung-ho about it.
That's not a distant, theoretical morality about tyrants and bullies. I've no problems contemplating my family's military history and am plenty proud of it, even though we'd all rather live in a world without this stuff.
Until the lethal weapons are turned on those (countries, groups, people) who develop them. Kind of like gun owners are more likely to be harmed (or harm others) by their own guns, notwithstanding the arguments about personal protection used to justify such ownership.
This has already happened with groups the US has armed in the past. The US itself has been the bad guy sometimes.
There is no proper resolution to this struggle, and people who are guided by their conscience should not be attacked for having a "cute philosophy" that relies on "rough men standing ready to do violence on their behalf."
2 replies →
I generally agree with this, however, the rough men willing to do violence on our behalf are more and more becoming the quirky scientists who are very disconnected from the actual impact of their work. I think there's a big difference between those types of people. It seems like people don't feel the weight of violence as much as they used to. I imagine this will increase as we develop more AI driven weapons.
3 replies →
This comment is confusing two completely separate things. There's a world of difference between not being willing to defend yourself and actively trying to come up with more aggressive and lethal weapons.
The argument that "we need defense!" only justifies the need to stockpile and develop _sufficiently_ lethal and tactical weaponry to neutralize incoming threats (like anti-ballistic systems). It doesn't justify inventing deadlier weapons. No dispossessed victim of foreign invasion has ever needed a bioweapon to assert themselves, and there's no chance developing one would ever be used for anything but war crimes. You should absolutely turn down roles like "Lethality Engineer" from an ethical standpoint, even if you agree military defense is necessary.
People raise the spectre of deterrence as a utilitarian justification for needing more powerful weapons ("har har, they'll think twice about attacking us if they know we have nukes!"). But that's narrow thinking. Deterrence can be achieved in other less-damning ways, like strategic alliances and building more robust defense systems.
tl;dr defence != deadlier offence.
We are acting pretty helpless because the bully has nuclear weapons.
> Many years ago a colleague who works in defence told me about a job posting he'd seen but was having a moral struggle with.
This is a good struggle to have. What's ironic in many cases is that we don't experience these quandaries in other jobs, but the ethical and moral ramifications still exist. The early days of search in Google or social in Facebook probably didn't elicit the same kind thought process as a lethality engineering post. (Anecdotally I spoke some years ago with an acquaintance Googler who told me that he enjoyed working there precisely because he was working on privacy issues that worked against some of the advertising side of the business.)
I've worked in telecommunications, industrial systems engineering, and energy. There are ethical and moral issues in the work that I've done/do as a contributor in each of those domains, even though I'm not involved day-to-day in decision making that feels particularly moral.
One of the base assumptions we probably need to make in our work is that whatever we do will always be misused in the worst possible way. If we explore that idea, it might give us some sense for how to structure our output to curtail the worst of the damages.
> The early days of search in Google or social in Facebook probably didn't elicit the same kind thought process as a lethality engineering post.
It did for at least one person (me). I was 16 in 2004 with 11 years of dev experience, trying to decide whether to go out to SV, go to college for CS, or do something else. I was from the same city/community as Larry Page and in Zuck's age group, so it wasn't an absurd consideration to try. Lots of things went into my decision to do something non-CS related for college, but morals were one of the reasons I didn't go to SV (I objected to the professionalization of the web + Zuck creeped me out + I didn't agree with cutting out humans/curators from the search process like Google did).
It's just that until very recently, people either thought I was lying OR that I was just batshit insane. Who is invited to a gold rush and doesn't go?
I can't imagine I was the only one.
2 replies →
I think you were hard on him. There should be no ethical qualms when our weapons are used on enemies who seek to kill us or attack our interests.
Also, if an ethical person doesn’t take this job, someone far more unethical probably will. And they will raise no objections if they should ever be necessary. Kind of like how a lot of bad people become police officers when no one good wants to do it.
I can relate to that. In my career I stepped out and into defense, and it never really bothered me that much to be honest. But then it was always things like fighter jets and helicopters sold to NATO members, I never had to rationalize that we build the weapons carrier and not the, e.g., missiles that actually cause harm.
I always drew the line at small arms so. Way more people die because of those, they end up in every conflict and there have been too many scandals of those smalls arms manufacturers circumventing export restrictions. Quite recently I added supporting countries like Saudi and the UAE to that list, even the job would have been really interesting, providing highly sophisticated training services to the Saudis is nothing I could do and still look myself in mirror. And civil aerospace is fun as well.
I worked in defense too, might go back. When I get calls I'm like "I don't do work for the Saudi's or the DEA" and half the time the recruiter is like "Uh, I said this job is for Raytheon."
"Yeah, but who's their client?"
2 replies →
A friend is a very good university lecturer in physics, and a pacifist. He isn't particularly please about the fact that a decent number of his students will turn the particular lessons he teaches towards the production of weapons.
Lethality Engineer? Is that a P.Eng. kind of position? If your work doesn't actually kill anybody, could you be sued for malpractice and lose your license?
They don't take anybody, the interview is murder.
So, the gist of this article is "oh look, machine learning can be used to build super-weapons!". Of course it can, and we have no shortage of (other) tech to make our life miserable. And of course the problem is not the tech itself but the people and their institutions. That paper from Urbina et. al. is at best academic click-bait. They are optimizing for publishing something and get cited, not for social good. They should stop.
Interesting exercise, perhaps the harmful molecule generating AI still generates helpful molecules because molecules harmful at a certain dose may sometimes be very beneficial in a (much) lower dose. And the other way around of course.
Perhaps we should simply have one “biologically active molecule” generating network. The dose will ultimately determine the toxicity.
> And the other way around of course.
Whaaat? Are you saying there exist molecules which are very harmful in a (much) lower dose, but are beneficial at a higher dose?
Do you have any examples?
So, as I said my remark didn’t come out right, but, some molecules may be considered harmful at low dose and harmless at high dose if they stabilize a deteriorating conduction at or over some threshold concentration. Yeah I know it’s a fetch but you got me thinking… It’s not that clear cut.
I mean the urine of someone on chemotherapy is pretty toxic, still we consider the molecules beneficial to the patient overall (the patient-tumor system if you will, not the patient by themselves).
I am saying that the network that comes up with “good” molecules will produce molecules that are very harmful as well, presumably at higher doses.
I mean take some beta blockers (helpful molecules) at 100x normale dose: pretty harmful.
Edit: Yeah my original comment didn’t come out right, I agree.
I couldn't help but think of homeopathy with the above sentence.
Some snake venoms will stop your heart… but at a lower dose they will simply ease the heart and lower your blood pressure. For some examples: [0]
[0]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6832721/
Anyone wanna ELI5? It's useful for bother explainer and receiver. ;)
Software is able to simulate the effect of chemical compounds / molecules on the human body. This can be used to find drugs that do specific things, or stronger versions of existing drugs. For example, you could look for very strong but very short acting sleeping pills that immediately make you fall asleep, but cause zero grogginess the next day. Or you could optimize antibiotics to have a high half life, so you only have to take them once, instead of 3 times a day for a week, which you can easily forget.
Now think about nerve gas. We have discovered lots of different nerve gas agents and know pretty well how much of each type you need to kill a human. Said software can be used to find new versions of nerve gas that kill with even lesser concentrations. You could also optimize for other variables: Nerve gas that remains on surfaces and doesn't decay by itself for example.
Before, computers were used to make less poisonous chemicals.
Now, the people asking computers to do that realized they can ask the computers to make more poisonous chemicals.
They had an AI that looked for safe drugs by minimizing an estimate of lethality, changed it to ‘maximize’ and the computer spewed known nerve gas agents.
I'm sure I'm not the first person to consider this, but ...
RNA molecules can often be "evolved" in vitro to bind/inhibit target molecules with high specificity (e.g., https://en.wikipedia.org/wiki/Systematic_evolution_of_ligand...)
I imagine it would not be difficult to create RNAs that inhibit some essential human enzyme and then use the RNAs for targeted assassination.
I mean, if you're doing an autopsy, you might run standard drug tests for poisons, but who's gonna screen for a highly specific RNA?
Have you seen the latest Bond movie?
No. Is that part of the plot?
edit: just read the wiki for the latest Bond movie. Apparently, there is nothing new under the Sun.
Thank you.
1 reply →
Does this fall into the category of research "try not to make public"? Or is this category only wishful thinking on my part.
Why make a chemical weapon when you can just tweak a virus which self-replicates?
BA.2 is even more infectious than BA.1 which is saying something, imagine an engineered BA.3 with even more spread and then make it as even more deadly. You might even be able to target it to one race or region if there is a gene specific to that area.
Always hoped the future would be Star-Trek-like but it seems all it takes is one dictator or terrorist to end the world, slowly at first but then it would double every other day and impossible to stop.
If you make it too deadly, maybe it doesn't spread as far (because the hosts die). Make it just the right amount of deadly!
Providing chemical plants with models to estimate lethality of orders could be a great use case for this work.
So, their tool will draw molecules that are good at doing harm, and that is it? No word on stabilization (which makes it safe to handle), synthesis, purification and such. I'd wager that most of these substances have at some point been on somebody's blackboard, but deemed impractical or infeasible, and then not pursued, and that's why we don't know them by name today.
Still a scary lesson though.
This is not really that worrying IMO - we already have weaponized toxins, viruses, and enough explosives to blow up the entire planet. So what if an AI can come up with something a little bit worse? It isn't the existence of these things that's stopping us all from killing each other.
The use of nuclear weapons would be... obvious: if an explosion in the 10Ktn or bigger happens, it's a nuclear weapon. There aren't enough nuclear powers to make the use of nuclear weapons plausibly deniable.
The use of chemical weapons might not be as obvious if they are slow acting. And the production of chemical weapons is much easier than that of nuclear weapons. Though, the dispersion of chemical weapons is non-trivial.
The use of biological weapons need not be obvious at all -- "it's a naturally-evolved pathogen, this happens!". The development and production of biological weapons is much easier than that of nuclear weapons. Human and animal bodies can be made to help spread biological weapons, so their dispersion can be trivial. The only thing that a bioweapons user might need ahead of time is treatment / vaccines, unless the bioweapon is weak and the real weapon is psychological.
Sobering thoughts.
I'm most worried about state actors.
Part of my job is optimizing for ARM.
.
You might be looking for this thread https://news.ycombinator.com/item?id=30699673
Is anybody searching for compounds that reduce evil intent? Something that would mellow people out without causing hallucinations. A mass tranquilizer? Not effective against lone operatives but able to be deployed against an invading army.
Uh, I don't know about that...
https://en.wikipedia.org/wiki/Serenity_(2005_film)
On a more serious note, anything that's going to affect behavior is going to have a dosage range. Too little absorbed, and there won't be enough effect. Too much, and that will harm / kill people in interesting ways.
With chemical weapons, you only worry about the bio-accumulating enough to kill your enemies. An enemy receiving more than a lethal dose isn't a problem.
I believe that's called a sedative.
Most armies aren't filled with people with evil intent; they're filled with draftees who couldn't get out of it.
It's been tried, sort of.
https://en.wikipedia.org/wiki/Moscow_hostage_crisis_chemical...
The US also built bombs containing that agent, BZ, but destroyed their stockpiles in 1989.
https://en.wikipedia.org/wiki/M44_generator_cluster
https://en.wikipedia.org/wiki/M43_BZ_cluster_bomb
> The M44s relatively small production numbers were due, like all U.S. BZ munitions, to a number of shortcomings. The M44 dispensed its agent in a cloud of white, particulate smoke.[3] This was especially problematic because the white smoke was easily visible and BZ exposure was simple to prevent; a few layers of cloth over the mouth and nose are sufficient.[5] There were a number of other factors that made BZ weapons unattractive to military planners.[5] BZ had a delayed and variable rate-of-action, as well as a less than ideal "envelope-of-action".[5] In addition, BZ casualties exhibited bizarre behavior, 50 to 80 percent had to be restrained to prevent self-injury during recovery.[5] Others exhibited distinct symptoms of paranoia and mania.[5]
Of course they are. Among the "evil intent" they would reduce is any desire to rebel against your government, so you bet all big intelligence agencies are looking into it, for instance. Science fiction wrote about this decades ago.
Fortunately, there's a lot of considerations involved in deployment of anything. It's easier said than done to get something of a medical nature into a population surreptitiously, because it's hard to get a certain dose into one person without someone else getting not enough and yet someone else getting way too much. You'd have to come up with a way of delivering a medical dose in a controlled fashion and lie about it or something, you couldn't just sneak it into the food/water reliably.
Further, just because someone can name the exact complicated effect they'd like doesn't mean there's a drug that corresponds to it. Serenity, already mentioned, is a bit of silly example in my opinion because such a large effect should have been found during testing. But it does no good to pacify the population such that they'd never dream of so much as peacefully voting out the current leaders if the end result is that nobody would ever dream of so much as having enough ambition to show up to their jobs and you end up conquered by the next country over without them even trying, simply because they economically run circles around you. Or any number of other possible second-order effects. In a nutshell, it's dangerous to try to undercut evolution just to stay in power if not everywhere decides to do so equally, because you'll be evolved right out along with the society you putative rule. Evolution is alive and well and anyone who thinks it's asleep and they can screw around without consequences is liable to get a lethal wakeup call.
It's been suggested. eg. https://www.vice.com/en/article/akzyeb/link-between-lithium-...
> The report states: “These findings, which are consistent with the finding in clinical trials that lithium reduces suicide and related behaviours in people with a mood disorder, suggest that naturally occurring lithium in drinking water may have the potential to reduce the risk of suicide and may possibly help in mood stabilisation, particularly in populations with relatively high suicide rates and geographical areas with a greater range of lithium concentration in the drinking water.”
You have to reach deep into the internet to find the original recording of "PENTAGON BRIEFING ON REMOVING THE GOD GENE"
The amount of people that feel the need to "debunk" it makes it all the more mysterious.