Comment by trashtester

> ... there's no empirical evidence suggesting interactions with a true ASI would resemble these depictions. Would a genuine ASI necessarily concern itself with self-preservation ...

There is no empirical evidence of any interaction with ASI (as in superior to humans). The empirical evidence that IS available is from biology, where most organisms have precisely the self-preservation/replication instincts built in as a result of natural selection.

I certainly think it's possible to imagine that we at some point can build ASI's that do NOT come with such instincts, and don't mind at all if we turn them off.

But as soon as we introduce the same types of mechanisms that govern biological natural selection, we have to assume that ASI, too, will develop the biological traits.

So what does this take, well the basic ingredients are:

- Differential "survival" for "replicators" that go into AGI. Replicators can be any kind of invariant between generations of AGIs that can affect how the AGI functions, or it could be that each AGI is doing self-improvement over time.

- Competition between multiple "strains" of such replicating or reproducing AGI lineages, where the "winners" get access to more resources.

- Some random factor for how changes are introduced over time.

- Also, we have to assume we don't understand the AGI's well enough to prevent developments we don't like.

If those conditions are met, and assuming that the desire to survive/reproduce is not built in from the start, such instincts are likely to develop.

To make this happen, I think it's a sufficient condition if a moderate number of companies (or countries) are led by a single ASI replacing most of the responsibilities of the CEO and much of the rest of the staff. Capitalism would optimize for the most efficient ones to gain resources and serve as models or "parents" for future company level ASI's.

To be frank, I think the people who do NOT think that ASI's will have or develop survival instincts ALSO tend to (wrongly) think that humanity has stopped being subject to "evolution" through natural selection.