Comment by qbxk

i think the "problem" here is that for all of human history we have always been able to use mastery of language as a signal for intelligence and competence, of which LLMs are neither. it's possible this is even instinctual it's so ingrained in our concepts of "other minds". so we're going to have to get used to the fact that just using language well isn't enough to prove intelligence, certainly not consciousness.

which then begs the the question, what is the magic ingredient, on top of use of language, that we have that bestows these qualities?

and also the observation that whatever this ingredient is, it must be very difficult to measure or prove which is maybe why we stuck with the crude, but easy to wield, "use-of-language" test for so long.