← Back to context

Comment by astrange

3 years ago

We already fail to align nonhuman intelligences (car companies) and they regularly kill people (by manufacturing increasingly large pickup trucks that office workers buy for an ego boost and then hit people with).

This doesn't wipe out the entire human race because nobody and nothing is capable of executing such a perfect plan because the real world contains something called entropy.

6 comments

astrange

Reply
pas  3 years ago

the argument is that the car company becomes something like China mixed with North Korea, big, powerful, uncaring, unavoidable, with a lot of resources, expansionist, etc.

car companies, however misaligned they are, are not particularly smart, nor are they generally intelligent. they are paperclip maximizers, but that's also their limitations (sell more cars). it took an eccentric madman to even open their eyes to a new untapped market (EVs!), of course we can argue that before Tesla car makers were in a metastable equilibrium, and incumbents were unable to rationally break out of it ... but that just shows how narrow their search space is.

these car companies are run by humans, regulated by humans, etc. they are pretty well aligned. it shows because we saw that they are just mimeing self-improvement. we know they were working on EVs, but very half-heartedly. (because they are risk averse, also because regulators don't let them merge into one giant company, etc)

  • astrange  3 years ago

    They didn't get into EVs because 1. Japan didn't like them (and still doesn't) because they have very expensive electricity 2. they were already getting into selling giant pickup trucks, not luxury sedans 3. China wasn't ready to compete yet.

    They are limited by participating in the economy, but actually that's the strange thing about this superintelligence scenario - it never seems to include the economy. In other words, if you invented an AGI, it would have to get a job to pay for its AWS credits.

    • NumberWangMan  3 years ago

      One interesting scenario I've heard is that of the "ascended economy", where more and more economic activity is simply to provide inputs into things that other businesses need, rather than what humans need.

      Like, a mining company produces steel, that is used to produce mining trucks, that is used by the mining company. Factories produce silicon, that are used to make solar panels and chips, for power and computing for the AI-run mining trucks. And at the root, AIs run everything, just trading money back and forth to buy the things they need and keep working.

      It's just a human economy, with the humans gone. Pointless economic activity.

      The question is, what's the path from here to there? Well, it probably looks like a multipolar trap, where every company has an incentive to automate more. Humans try to stop or slow it down, but companies with more AI can hire more lobbyists and get things done. AIs are smart enough to think of all the things that could slow their business down, and get rid of obstacles in the same way humans do. There are probably still lots of small, inefficient human owned businesses, but fewer and fewer over time. (We've seen this trend for a while already!)

      I'm hoping a massive backlash would stop this -- but if it's coupled with short-term increases in living standards for people, I think it would have a lot of support, until things start to go really bad, but by then it might be too late, if AIs have control over news media and telecommunications and can stop dissenting humans from coordinating any sort of rebellion.

      2 replies →