Comment by sp1nningaway

I like this bit from the wiki article:

Speaking through Zarubin, Dneprov writes that "the only way to prove that machines can think is to turn yourself into a machine and examine your thinking process", and he concludes, as Searle does, that "even the most perfect simulation of machine thinking is not the thinking process itself."

That feels similar to what Chiang is saying. The physical state of being a human is part of human consciousness. An LLM would require many more "levels" that haven't been achieved yet. An LLM with a body, sense organs, physical tools, social dynamics, etc. would all be steps on the path to a "conscious" LLM.