Comment by sequoia
2 years ago
I'm disturbed by the idea that an AI could be used to make decisions that could proactively kill someone. (Presumably computer already make decisions that passively kill people by, for example, navigating a self-driving car.) Though there was a human sign-off in this case, it seems one step away from people being killed by robots with zero human intervention which is about one step away from the plot of Terminator.
I wonder what the alternative is in a case like this. I know very little about military strategy-- without the AI would Israel have been picking targets less, or more haphazardly? I think there may be some mis-reading of this article where people imagine that if Israel weren't using an AI they wouldn't drop any bombs at all, that's clearly unlikely given that there's a war on. Obviously people, including innocents, are killed in war, which is why we all loathe war and pray for the current one to end as quickly as possible.
> B., a senior officer who used Lavender, echoed to +972 and Local Call that in the current war, officers were not required to independently review the AI system’s assessments, in order to save time and enable the mass production of human targets without hindrances.
> “Everything was statistical, everything was neat — it was very dry,” B. said. He noted that this lack of supervision was permitted despite internal checks showing that Lavender’s calculations were considered accurate only 90 percent of the time; in other words, it was known in advance that 10 percent of the human targets slated for assassination were not members of the Hamas military wing at all.```
So, there was no human sign-off. I guess the policy itself was ordered by someone, but all the ongoing targets that were selected for assassination were solely authorized by the AI system's predictions.
This sentence is horrifically dystopian... "in order to save time and enable the mass production of human targets without hindrances"
Hm OK, I read this a bit differently. I read these sections:
> One source stated that human personnel often served only as a “rubber stamp” for the machine’s decisions, adding that, normally, they would personally devote only about “20 seconds” to each target before authorizing a bombing — just to make sure the Lavender-marked target is male.
> According to the sources, the army knew that the minimal human supervision in place would not discover these faults.
I took this to mean that a human did press the "approve" button on the computer's recommendation. Though they make clear they were basically "rubber stamping" the machine recommendation.
But to my point:
> “There was no ‘zero-error’ policy. Mistakes were treated statistically,” said a source who used Lavender.
What is the "zero-error" alternative approach for dropping bombs in a war, or firing rockets for that matter? I don't understand the implicit comparison between this approach to targeting and a hypothetical approach that allows war to be waged without any innocents dying or buildings being destroyed. This system should be compared to whatever the real alternative is when it comes to target selection. Again I know nothing about military strategy, I'm hoping someone with more experience will speak up.
To use an analogy: if we are talking about self-driving cars, the rates of collision or death should be compared the rates of collision or death in cars driven by humans. Comparing against some imaginary scenario where cars have no collisions and cause no deaths doesn't make sense.
> What is the "zero-error" alternative approach for dropping bombs in a war, or firing rockets for that matter?
Honestly, I'm not sure. Obviously humans make errors of all sorts as well, and even make intentionally unethical decisions.
I think the horror of this situation is that it makes war easier to wage. Accepting that all war has costs measured in blood, we should want less war. However, those in control of military forces always have incentive to wage war, so removing friction from the process is dangerous.
Off-topic of AI, but on-topic of your question:
The actual alternative to unleashing AI assassination is not human-selected targets, but not waging war. It isn't necessary to destroy Hamas with violence, it would have worked better to give Palestinians dignity and self-determination long ago. That can still work, although until it does Hamas will continue to be a problem. But as I said, war is useful for the political leaders of Israel, so they stoked and fed the flames for decades to maintain an excuse for the war machine.
62 replies →
>> Comparing against some imaginary scenario where cars have no collisions and cause no deaths doesn't make sense.
That's not the whole story. For example, we ban certain kinds of weapons -cluster munitions, chemical weapons, biological weapons, ideally we'd ban bloody mines- not because they kill too many people compared to "conventional" weapons (they don't) but because they are considered especially ... well, wrong, in the moral sense.
So maybe we decide that being killed by a machine, that decides you're a target and pulls the trigger autonomously is especially morally wrong and we don't accept it.
8 replies →
The difference is between inaccuracy of a weapon hitting a target and inaccuracy of target selection in the first place.
Remember the scene in Men In Black where the recruita do target practice? They were all accurate at hitting what they shot at but only Will Smith's character was accurate at selecting a target. This AI chooses targets; it does not fire weapons.
10 replies →
I think 90 percent accuracy in this case means 10 percent of "suggested terrorists" were overturned with detailed human review. There's no way the Israelis were actually able to reliably question the Gazans about whether they really were terrorists.
So the issue isn't that there's errors, it's that the army knows there are errors and expect humans to pick them out in 20 seconds- which they know realistically won't happen. The human only has two realistic choices- approve every target, or disapprove every target (which gets you reassigned to another role).
It's the classic statistics case of two medical diagnostics for an underlying value that isn't directly observable.
> > “There was no ‘zero-error’ policy. Mistakes were treated statistically,” said a source who used Lavender. > > What is the "zero-error" alternative approach for dropping bombs in a war, or firing rockets for that matter? I don't understand the implicit comparison between this approach to targeting and a hypothetical approach that allows war to be waged without any innocents dying or buildings being destroyed. This system should be compared to whatever the real alternative is when it comes to target selection.
I think you've misunderstood the "zero-error" statement. It's not saying "there must be zero errors", rather that "errors don't exist - only some level of collateral damage". Hence the follow up about things being viewed statistically.
They view it in the same way that you suggest they should - that there will always be deaths and the questions is whether the system leads to more or less of them.
Personally I view that as a very utilitarian argument when applied to a machine of war. It embeds the concept that some loss of innocent life is acceptable.
We also have to be open to the possibility that Israel is committing a genocide and the goal is to kill as many Palestinians as possible and terrorize the rest. That the AI system’s main purpose isn’t to be accurate in selecting target, but rather to manufacture a reason to kill more Palestinians than a human ever could. Another function could be to remove accountability from a targeting officer. Zero-error is never really a desired feature, in fact zero-error would be a bug, as it would prevent the genocide being conducted efficiently.
What we may be witnessing is the first information age level genocide, where the killing is done at the behest of a statistical function with near infinite computing power.
5 replies →
Later in the article they talk about how they specifically approved up to 15-20 civilians to die with those marked individuals and would bomb their homes as a first option.
I’m disgusted by this, I don’t care anymore what happened in October, this needs to stop. Israel government cannot be trusted to run this war, it’s turned into genocide and we’re all complicit letting them do it and supporting them. I can’t believe people actually support this, it’s clear they’ve forgotten Palestinians are people.
9 replies →
[flagged]
13 replies →
I imagine you get to tune the probability window of "person is >90% likely a Hamas terrorist" and choose how many innocent people you kill. Who set the window?
"Hamas terrorist" criteria: a male of fighting age, give higher weight to those congregating with others of fighting age. Basically take out a generation of Palestinian men and you're all set. Lovely.
>This sentence is horrifically dystopian... "in order to save time and enable the mass production of human targets without hindrances"
Reminds me of similar industrial thinking of a certain previous fascist government.
A war that would only kill 10% civilians would be a massive improvement over any recent conflict.
17 replies →
>Basically take out a generation of Palestinian men and you're all set.
Now that we've established that this is horrific, please turn a small portion of your attention to American predictive policing systems (digital and not) and the circumstances that lead to mass incarceration (including the War on Drugs).
...and then target them at home along with their entire family.
3 replies →
I don’t like Lavender. I think humans should always be in the loop. I’d like to see more care by analysts for kill orders.
That said, any organization might do something if it’s 90% accurate. Assuming it even is (doubt it), I think any fair evaluation of such a technology must ask:
What is the accuracy of inexperienced humans in the same position who are rushing through the review during a blitz invasion? If they have battle experience, what about them, too? (I’m assuming most won’t.)
Is the system better than those humans or worse? How often?
Do the strengths and weaknesses of the system allow confidence scores on predictions to know which need more review? Can we also increase reviews when the number of deaths will be high?
That’s how I’d start a review of this tech. If anyone is building military AI, I also ask that you please include methods to highlight likely corner cases or high-stakes situations. Then, someone’s human instincts might kick in where they spot and resolve a problem even in the heat of war.
Lavender AI in it's current form is not a technology that should be considered at all for anything close to this. It's incompetent.
Basically AI is being used as a wheel-of-death and nothing more.
Anyone believing differently in my opinion is both delusional, and complicit
It is very clear to me that that is a sentence reflecting the editorial interpretation of the paper rather than a direct quote. You might agree with the interpretation - I think I might - but that is very different from this specific sentiment being something Israeli leadership are openly saying.
90% is a BS number . Computed basis what ? What is the baseline how did they benchmark . Is there any data whatsoever to back this claim ?
They just spout a high number that is not 100% (clearly civilians are being killed publicly undeniably ) claiming 100% would be too obviously ridiculous.
More than half of 32,000+ (more under the rubble) killed are woman and children, Hamas is still quite able to fight, hardly any hostages has been recovered .
Israel labels any sort of civilian organization as hamas including journalists, medical and aid staff. 200 UN staff and 100 journalists are dead so far . Israel’s argument is UNWRA terrorist aiding and journalists were also secretly Hamas and doing non journalistic stuff when killed so they include them in legitimate targets .
If you consider everyone is Hamas unless otherwise proven then 90% is possible .
There is no realistic way an algorithm was designed factoring in the level of destruction of infrastructure never seen in any real world data and also benchmarked accurately.
[dead]
> I wonder what the alternative is in a case like this.
It seems obvious to me that the alternative would be a slower process for picking targets leading to fewer overall targets picked and the guarantee that a human conscience is involved in the process.
Or alternatively pressure from the top down on targeting specialists to get more and more targets selected resulting in less quality and effort spent on selecting targets and maybe leading to rubber-stamping proposed targets without adequate consideration. Which isn't to suggest that that would definitely make the AI better per se
It's an army too cowardly to have dismounted infantry protecting their tanks, so instead their conscripts burn alive in there when they get in contact with actual militants.
It's an army incompetent enough to recreate the rubble of Stalingrad to help its enemy.
How would they go about producing officers that could enact such pressure? How would they recognise the difference between a specialist and a charlatan whos family is good friends with the army rabbi?
3 replies →
Look I know this is gonna sound cliche but the thing they should do is not engage in an offensive asymmetrical war and bomb a dense urban area full of innocents for basically no reason. Then they wouldn’t need the little ai.
This is obviously veering way off course of the topic of AI at this point, but I imagine the residents of kibbutz be'eri and the 100+ hostages still held in Gaza would disagree that Isreal is fighting for "basically no reason." I'm interested in analysis and criticism of Israel's use of AI in this case but suggesting Israel has no causus belli is absurd.
I never understood why the argument that only war can bring hostages back, or that it is the most reliable way, was allowed to be propagated in public for so long.
> but I imagine the residents of kibbutz be'eri and the 100+ hostages still held in Gaza would disagree that Isreal is fighting for "basically no reason."
On the contrary, I think that those exact people would agree the most. Do you think that they do not wish that Israel did a hostage exchange instead of starving and bombing them together with their captors? To bring the "low hanging fruit" example, do you think that the three hostages who were waving white flags nearly entirely naked, and who were subsequently murdered by the IDF; do you think that they or their families prefer(preferred) this devastation that lead to their deaths instead of a simple hostage exchange?
What do you think would happen if IDF killed most of Hamas and had their last few forces cornered with no escape, and were getting close to them? Do you think the hostages would not be killed by either their captors or by IDF as collateral damage in such a scenario?
Claims that the systematic destruction of Gaza and genocide(-lite?) serves the goal of bringing back the hostages is such an obvious cover for bloodthirst that it is honestly intellectually-insulting to keep reading it over and over again.
11 replies →
OP didn't say Israel is fighting for basically no reason.
You're twisting their words, I'll assume out of a misreading. Read the comment again. They clearly said that there's no good reason to bomb Gaza the way that they have been doing, resulting in the murder of thousands and thousands and thousands and thousands of civilians.
13 replies →
Disturbing indeed. I've been worried a push back in AI is coming and this sort of story could be a tipping point and certainly would justify a period of reflection.
And your probably right that the alternatives maybe worse, the folks behind Lavender could probably even prove it with data.. but there should be a moral impetus to always have a human in the loop regardless. And any such attempt to justify it won't capture the publics attention like a sky-net doomsday happening over the civilians in Gaza.
there should be a moral impetus to always have a human in the loop regardless
I don’t understand how to come to this. War is crap, not a dinner party. There’s always a human on both sides who will drop a bomb and laugh on camera, with no responsibility. Go watch it (actually don’t, it’s NSFL). Reading this thread feels like everyone watched and believed in that movie where they tried to select and eliminate a target for 2 hours with futuristic hi-tech. A human hesitates to press the button before the war. When in it, he will only be concerned with things like ammunition saving and tactical nuances. There’s not much more morals in a human who usually sits there at the button than in AI automation.
The thing that is different is now that human has an excuse: "The computer told me to put them in the oven."
There's an old IBM presentation going around, from 1979, that says "A computer can never be held accountable, therefore a computer must never make a Management Decision." We know that humans make monstrous decisions in war; many of us remember seeing the Collateral Murder video, and everyone has at least heard of the Nuremberg trials. When humans make monstrous decisions, at least some of them, sometimes, hang for it. The computer here serves mainly to diffuse responsibility for decisions that would be made in any case. Who will hang?
It's a good question. I also immediately found myself asking the same one of myself after posting that comment. I guess part of me just wants as many possible breakpoints along the process as possible.
But also at least then you have someone who is liable when things go wrong. When its fully automated, like the other comment mentions, they can just shrug and blame the AI. Who gets sued when a self driving car kills someone by accident? I don't know. Perhaps a lack of ownership is excusable. But when a weapon deliberately kills someone I think we need to have ownership somewhere.
Perhaps as a general rule the maker of the AI system should have liability for the AI up until someone else signs and accepts that responsibility. None of this "Company does not accept liability" crap. They have to make it clear that "customer accepts liability" or else it's them. That way they will be incentivized to make the military or whoever sign.
I would strongly argue that even being able to prove that a human might perform worse is not an acceptable excuse for the reasons I will outline. The bar for a computer needs to be significally higher than that of a human.
We know that humans can make mistakes, due to a multitude of reasons. They can be tired, moody, distracted, stressed out, time-pressured, simply not care enough, etc, all contributing to making the wrong call. But a computer does not suffer from such issues. Secondly, a computer (program) is able to perform billions and billions of computations within some time period in order to ENSURE that doing this thing with grave consequences is absolutely warranted.
Maybe for some domains we can tolerate errors from AI, but when deciding whether a person (and everyone around them) lives or dies, surely simply being on average even more accurate than a human is not enough. "Killbots" MUST be extremely heavily regulated.
Pushback on AI will of course have a “National security” exception. If the industrial level facial recognition tech in Xinjiang was forgotten I doubt this will make a difference
>I wonder what the alternative is in a case like this.
Don't Create The Torment Nexus
I think that once you start from the viewpoint that you're not going to create the Torment Nexus, it becomes a lot easier to avoid creating the Torment Nexus.
"A computer can never be held accountable, therefore a computer must never make a management decision"
The IDF only read the first half of the classic IBM slide!
There would have been slower target selection.
A lot of news around the bombing called out the uniquely large scale and rapidity of the campaign.
This was a preview of future conflicts.
We're entering the WWI phase of new technology being brought without rules to conflicts where the abuses will be horrific until rules are finally put in place.
We crossed the line of machines that automatically kill a long time ago. A heat seeking missile, or a shell that detects and target tanks [1] are effectively doing that. Software selects the target. The soldier only points in the general direction. AI is only a small technical increment.
[1] https://en.wikipedia.org/wiki/SMArt_155
But you know soldiers are in a tank, and you know a pilot is in a plane. Who's in an apartment?
It's never really that clear-cut, though. Human drone operators, pilots, etc. routinely send missiles into cars, buildings, weddings, etc. that cause collateral damage, killing or maiming innocents and passers-by. Sometimes it's an accident, but not always.
And that's just when we even try to limit damage, vs indiscriminately firebombing or nuking entire cities.
We shouldn't demand perfect accuracy of AI when we don't expect the same of humans. Long ago, we decided collateral damage in war is acceptable, especially when you end up winning the war and there's nobody left to prosecute you except historians =/
[dead]
Difference is someone has to fire those weapons and that someone is responsible.
This system bassicaly just gave everyone a score from 1 to 100 of how luckely they are part the military wing of hamas.
Another system would signal that target is at home and it's time to bomb. This system was using phone to geo-locate and due to nature of living in Gaza phones transfer hands often.
Without Lavender they would have dropped less bombs IMO.
Is having a human make those decisions really better? It was humans who ordered the Holocaust, My Lai, Wounded Knee, Rwanda, Tiananmen, etc.
At least AI pretends to look at some data instead of just defaulting to tribal bloodlust... who's to say it can't be more ethical? It doesn't take much to beat our track record.
I think people are worried no one really understands how AI picks the target.
Reminds me of that story from probably 5-7y ago. Someone wanted to use AI to classify photos of tanks as soviet vs US. So he went to a US tank museum and took lots of pictures of the tanks under every angle. Did the same in a soviet tank museum. The resulting model worked great on that training dataset. Then he tried on photos outside of the training dataset. Turned out that it was cloudy the day he visited the US museum and sunny for the soviet museum, and the model used the color of the sky to classify.
An eternal story; I heard the same thing at university 22 years ago, except then it was NATO taking nice crisp in-focus photos of their own tanks from close up, while the images of Soviet tanks were all blurry and grainy because they came from high-altitude spy planes.
(This kind of human model hallucination is how and why I think Genesis got written and taken seriously).
https://gwern.net/tank
Seems like segmentation would be a better approach to identity objects in a photo rather than various other features.
> I think people are worried no one really understands how AI picks the target.
Yeah, I mean, black-box murder is never really desirable... but is it fair to assume AI will never be able to elucidate its reasoning? And that also seems a bit of a double standard, when so many life-and-death decisions made by humans are also not entirely comprehensible or transparent, either to the general public or sometimes even to the other individuals closest to the decision-maker.
Sometimes it's a snap judgment, sometimes it's a gut feeling, sometimes it's bad intel, sometimes it's just plain "because I said so"... not every kill list is the result of a reasoned, transparent, fair and ethical process.
After all, how long have Israel and Hamas (or other groups) been at each other's throats, with cries of injustice and atrocities about either side, from observers all over the world? And it wasn't so long ago we destroyed Afghanistan and Iraq, and Russia is still going at it because of the desires of one man. AI doesn't have to be perfect to be better than us.
If there's one thing humans are really, really bad at, it's letting objective data overrule our emotional states. It takes exceptional training and mental fortitude to be able to do that under pressure, especially life-and-death, us-vs-them pressure.
Humans make mistakes, too, and friend-or-foe identification isn't easy for humans either, especially in the heat of battle or in poor visibility. Training for either humans or AI can always be improved, but probably will never reach 100% accuracy.
Maybe we should start putting some hypothetical kill lists in front of both humans and AI, recording their decisions, and comparing them after a few years to see who did "better". I wouldn't necessarily bet on the humans...
What if the AI was trained on data collected and assembled by someone with "tribal bloodlust".
What happens when you put a computer in front of the judges at the ICC?
They ask an assistant for help?
Run it through some panel of experts and demand algorithm changes?
Send it to some Judge API and get back some JSON?
I dunno, what?
They're not exactly very good at preventing or punishing human atrocities, either... it's more of a symbolic group, or a tool of the victors, than anything resembling actual justice. I'd argue textbook authors have more of a lasting ethical impact than the ICC.
1 reply →
I'm having to take a few deep breaths before responding to some of these comments. The difference is Accountability. A computer can't be held accountable and a person can. Full stop. It makes all the difference.
The person operating the computer or building the AI could be held accountable though.
2 replies →
Perhaps it performs sentiment analysis of your social media posts.
So that's why I'm still alive. Hi, robo-overlords! Sarah Connor sux. Save me for last!
when a computer program designed by a human "makes" the decision, humans can claim that it was "a funny mistake", it was not their fault and pretend to be very sad for it.
Having a human to make those decisions is better because this human can be judged if commits war crimes or genocide or violates international war laws.
A computer can't be jailed and this is the real power of designing this system. To hide the criminals on a black box so nobody can be made responsible
Exactly my thoughts, the AI shields all responsibility from the humans.
[flagged]
[flagged]