Comment by js8

I think the biggest flaw of the Chiang's argument is the assessment that it's unplausible we have built consciousness by accident.

Remember that LLMs can do logic and reasoning came as a surprise to everyone; and for the same reason, nobody expected "next token predictor" trained on huge amount of data to evolve in this way.

But for the same reason, we cannot easily dismiss we didn't evolve (I mean by training an LLM, it's a form of evolutionary computation) consciousness as well. Our own consciousness (and reasoning and morality) might be an evolutionary consequence of "just trying to predict the world" as well.