Comment by sedan_baklazhan

It's not a guidebook, it's a thought experiment on "what if you could do that", and that's the entire point.

We can't expect to succeed, but our cycle from the ancient Greeks thinking there were four elements where the right mix of air, earth, fire and water would create any substance and thus it was possible to turn lead into gold, took us on a path that developed into alchemy, then chemistry, then physics, giving us at first far more elements, then we realised the name "atom" (Greek "ἄτομον", "uncuttable") was wrong and those were made of electrons, protons, and neutrons and the right application of each would indeed let us turn lead into gold…

And the cargo cults, clear cutting strips to replicate runways, hand-making their own cloth to replicate WW2 uniforms, carving wood to resemble WW2 radios? Well, planes did end up coming to visit them, even if those recreating these mis-understood roles were utterly wrong about the causation.

We don't know the necessary and sufficient conditions to be a mind with subjective inner experience. We don't really even know if all humans have it, we certainly don't know which other species (if any) have it, we wouldn't know what to look for in machines. If our creations have it, it is by accident, not by design.

seems like you are obsessed about the lack of the definition of the consciousness. But buisness doesn't need to understand what it is to exploit it

like with LLMs, we don't know what the "consciousness" is and if they have it, but it doesn't matter, we use them

> you can't copy something you have not even the slightest idea about

imagine if we had a way copy or emulate every cell and connection in the brain? we don't need to know which part of the brain is responsible for what, it would function even if we were still clueless about how it works

You might argue that you cannot really copy such complex stuff without understanding. But humans managed to copy creatures without understanding what each gene does.

I mean we already do 'it'-- by it I don't mean uploading people, but rather create businesses that operate people via an API then hook those APIs to profit maximization algorithms with little to no regard for their welfare. Consider Amazon's warehouse automation, door dash, or uber.

Of course it's much more extreme when their entire existence and reality is controlled this way but in that sense the situation in MMAcevedo is more ethical: At least it's easy to see how dangerous and wrong it is. But when we create related forms of control the lack of absolute dominion frequently prevents us from seeing the moral hazard at all. The kind of evil that exists in this story really doesn't require any of the fancy upload stuff. It's a story about depriving a person of their autonomy and agency and enslaving them to performance metrics.

All good science fiction is holding up a mirror at our own civilization as much as it is doing anything else. Unable to recognize ourselves we sometimes shudder at our own monstrosity, if only for a moment.