Comment by icepush

6 hours ago

Those are things that pose potential harm to great fractions of humanity (multiple billions of people) but none of them poses any threat to the actual extinction of all humanity.

This is not correct. Please go through each one again, taking special note of global warming and nuclear weapon development .

  • Why do you think global warming could kill literally every human being? What's the scenario in which that happens?

    • Runaway warming leading to a hot "supergreenhouse" climate was theorized. Alterbatively a snowpiercer scenario creating a snowball earth.

      Or, once billions of people die from climate change, planetary wars will start on the last hospitable areas, leading to the end of humanity.

      In contrast, AI ending "humanity" is limited by our own material power, would likely be stoppable, and would not easily reach areas isolated from technology

      2 replies →

    • There's a lot to unpack with this, and I am not an expert, but: crop failure across industrial monocrops leads to food shortages, wild fires destroy significant areas which support pollinator ecologies, fresh water become contaminated by rising sea water and blighted by drought, and perhaps most significantly, as resources become strained, humans become desperate and armed conflict proliferates. This is just top of mind.

      3 replies →

    • Worst case is warming oceans create a hypoxic environment where anaerobic bacteria thrive en masse and generate H2S over country-scale areas undersea. The chemocline breaches the surface and the ocean and atmosphere become poisonous to most complex life and agriculture, also stripping the ozone layer in the process, irradiating the surface. This is one mechanism posited for the end-permian mass extinction that eradicated most ocean and surface life, including the trilobites.

      Not sure how long we'd survive such a scenario, even sheltering underground. But surely it couldn't happen to us.

      (However, it now seems like the AI might get us first.)

    • Global warming causing war between nuclear armed opponents. Migration flows and competition over ever shrinking resources.

      It's silly to treat these things as separate categories as if only one will happen at a time. These issues are happening, and are going to happen all at once. That is the fundamental challenge. AI plays into that because it can make wars and nuclear exchanges so much more effective.

      If an AI tells the President the USA can survive China's nuclear barrage mostly unschathed, perhaps he'll press the button...

  • Global warming is happening at a slow enough pace that there is plenty of time for people to move to the poles/into caves/beneath the ocean/whatever. Nuclear war would be much more destructive in a much shorter range of time but you will have pockets.

    • People already entrenched near the poles will fight to keep others out. Right-wing anti-immigrant parties are on the rise right now already.

      They also tend to not want to recognise climate change -- because that would imply that people could migrate because of it.

    • Citation needed for evidence of human cooperation at this scale. As you say, climate change is slow moving, and there's no real evidence across the last 60 years of knowing about it suggesting humans are willing to do what it takes to survive it.

      4 replies →

  • How is global warning gonna kill all of humanity? Even a full scale nuclear war wouldn't manage that.

    They could destroy most civilisations, culture and scientific achievements though.

    • Because nuclear war balances itself: more nuclear bombings = less nuclear weapons, less people to send them. It will naturally stop at some fraction of humanity left which is incapable of making and launching more nukes

      1 reply →

    • In the exact same way major sudden changes in the climate have lead to the extinction of the majority of plants an animals over this planets history. We are not special.

      15 replies →

  • Humans are voluntarily driving themselves to extinction through sub-replacement birth rates worldwide. That's going to play out far quicker than global warming or pretty much anything other than every country on earth launching nukes at each other will.

    • Hard disagree here, population growth is still happening and will keep happening for some time, maybe step out of your bubble if you don't see that.

      Africa is still growing massively for example, world is not just western civilization. Sure at that rate and incerase of living conditions for everybody maybe in 1000 years population will be smaller, but its not that hard to fix if wanted - people used to have 10-15 kids as default.

Neither does an LLM model.

  • Oh indeed, but an LLM model that's reasonably smart could brute force find other training algorithms that are that dangerous. Just as it solves maths problems.

    This is the explicit plan of OpenAI and Anthropic.

  • LLMs can already control robots. Today's LLMs are inept at it, but they can do it.

    Things will be bad, unless someone does something to stop it. Will anyone do that? We don't know.

    https://xkcd.com/2278/

Arguing whether "all" or just "most" of humans dying is certainly a choice.

Survival of the species isn't enough.

  • In prioritizing risks I do feel it's a genuine qualitative difference though.

    Number of humans going to 1 million would be huge catastrophe, but after a few thousand years it gets back up to billions of people, renewed every generation.

    Number of humans going to 0 means that's it. One scenario has many orders of magnitude more missing humans, when you count future lives.