← Back to context

Comment by epistasis

3 days ago

There's a huge gap between what's being described in that comment and an "extinction level event" is that's what ELE means.

What's being talked about is a massive extension of human capability, into the intellectual sphere rather than the physical environment. The atom bomb had a clear way to get to extinction, a sudden chain reaction that leads to collapse of the entire biome.

The only way AI gets there is something like control of the nuclear weapons on all sides and that control becoming suicidal. Every other major event leads to at most civilization collapse, after which I assume we will have some sort of Butlerian Jihad restrictions on getting into the same situation.

But, I would love to know what I'm missing here, if ELE has clearer routes!!

5 comments

epistasis

Reply
lelanthran  3 days ago

> The only way AI gets there is something like control of the nuclear weapons on all sides and that control becoming suicidal.

Not the way I was thinking off; I was thinking more along the lines of brain atrophy until humans are helpless (maybe 2 complete generations of humans).

And that's a short extinction; the KT extinction took place over thousands of years. If, 1000 years from now, we are finally at the place described in "The Machine Stops", only then will an observer be able to decide that we don't have the necessary critical mass of intelligent humans anymore.

Humans got to be the apex predator simply on intelligence alone. It's possible that without thinking ability we'd go extinct.

  • epistasis  3 days ago

    > Not the way I was thinking off; I was thinking more along the lines of brain atrophy until humans are helpless (maybe 2 complete generations of humans).

    That's not how humans work. We developed math, art, literature, etc., because people love the act of doing it. We will continue to do that.

    We're not going to stop thinking because we don't have to. We stop thinking because we subject ourselves to political propaganda, or religious belief systems, or other group control methods that turns off critical thought. But even then, it's only a subset of the population, it won't get us all.

    • lelanthran  2 days ago

      > That's not how humans work. We developed math, art, literature, etc., because people love the act of doing it. We will continue to do that.

      Sure, for art, literature, etc. For math, physics, biosciences, etc, we pay people to do it. There are too few people who would work for free to push progress in those areas. We are most certainly not going to do that, we're going to leave it to the AIs and hope for the best.

      This sort of thing (human brains atrophying over generations) is like something out of a dystopian nightmare[1], but hey, that's natural selection for you: if intelligence is not a selection criteria (and it won't be if machines do all the thinking), it's unlikely that it will be present in future generations.

      -------------------

      [1]. "The Machine Stops", E. M. Forster, 1907. Or "Pump 6" (can't remember the author, Italian author). Just because it's fiction does not mean it's impossible.

  • keeda  3 days ago

    I actually think the opposite… it takes all my skills as an engineer to steer these things effectively and I’ve learnt more than I imagined I could in a very short period of time. And I don’t even use AI as heavily these days due to other commitments. This also aligns with how heavy AI users report feeling drained after intense coding sessions.

    My take is that after a period of turmoil, most humans will be doing higher level cognitive work than we’re doing now, because all the lower level work will be done by AI agents, and so progress will accelerate multifold.