Comment by throwaway_ab
3 years ago
It's not about super intelligence leading to kill all humans by default, but among thousands of emerging super intelligences it only takes one to act against humanity to kill all humans, and that is the worry.
3 years ago
It's not about super intelligence leading to kill all humans by default, but among thousands of emerging super intelligences it only takes one to act against humanity to kill all humans, and that is the worry.
I don't think that is necesarily so. If it's only one among thousands then we could use the others to fight it.
That’s a perfectly fine outcome. We already have that, since the incidence of psychopathy and mental illness is much higher than one in a thousand. A multipolar world with 999 well adjusted moral super intelligent AIs for every 1 problem case is a perfectly good outcome.
Bostrom et al are in fact arguing that near 100% of all intelligences will be unaligned by default and end up killing, enslaving, or otherwise neutralizing the entire human race.
We already fail to align nonhuman intelligences (car companies) and they regularly kill people (by manufacturing increasingly large pickup trucks that office workers buy for an ego boost and then hit people with).
This doesn't wipe out the entire human race because nobody and nothing is capable of executing such a perfect plan because the real world contains something called entropy.
the argument is that the car company becomes something like China mixed with North Korea, big, powerful, uncaring, unavoidable, with a lot of resources, expansionist, etc.
car companies, however misaligned they are, are not particularly smart, nor are they generally intelligent. they are paperclip maximizers, but that's also their limitations (sell more cars). it took an eccentric madman to even open their eyes to a new untapped market (EVs!), of course we can argue that before Tesla car makers were in a metastable equilibrium, and incumbents were unable to rationally break out of it ... but that just shows how narrow their search space is.
these car companies are run by humans, regulated by humans, etc. they are pretty well aligned. it shows because we saw that they are just mimeing self-improvement. we know they were working on EVs, but very half-heartedly. (because they are risk averse, also because regulators don't let them merge into one giant company, etc)
5 replies →
> A multipolar world with 999 well adjusted moral super intelligent AIs for every 1 problem case is a perfectly good outcome.
Such simplistic analysis is naive. A schizophrenic person can't spread, multiply themselves to increase its strength, but a rogue AI can.
The incidence of cancer cells is procentually very small, yet they routinely kill people, because they spread quickly.
The AGI and ASI instances that we have (e.g. GPT-4), require entire data centers with highly specialized hardware to run. How would a rogue AI replicate itself? You are worried about a fictional threat scenario that doesn't map to the real world.
1 reply →