Comment by Brian_K_White

I don't know about anyone else, but I am definitely not concerned with the mechanics, in the sense that a consciousness could be implimented in anything. There is nothing magic about biology, go ahead and Ship of Theseus every biological construct and sub process with some analog made out of other materials or even pure energy and the result is still the same consciousness. And I do not believe in any kind of actual soul in the religious sense.

That does not mean there is no difference between what conscious beings do, and what any mechanistic process does. Mechanistic does not mean "made of electrical signals" or made of anything in particular. A purely imaginary algabraic equation is not made of anything, yet is a mechanistic process. A thought is either made of nothing or made of biology depending on how you wish to think about it, yet is not a mechanistic process.

Even though a consciousness can also perform a mechanistic process that looks the same from the outside. An axle can turn because an electric motor turns it, or that same axle can turn the exact same way because you turned it. There is a purely exterior effect that is identical in both cases. Put the motor in a box with only the shaft sticking out, and put yourself inside the same box so the outside observer can only see the box and the shaft. Since everything is the same from the outside, I guess that proves that electric motors are conscious. They decide to turn shafts for internal reasons not all that different from the reason you decided to. Or it proves that neither the motor nor yourself are conscious or thinking.

It is unutterably stupid to confuse a person with a painting of a person. LLMs are nothing but paintings of people. People wrote everything it spits back out, and the mixing that it does is entirely explicable and reproduceable by plain mechanistic process.

Take all the words and write one each onto ping pong balls.

Add slightly different weights to the different balls so some are heavier than others.

Add slightly different magnets to each, so that some are slightly more attracted or repelled to others.

Change the shapes of the balls so that some fit up against others better than others.

Glue together a few balls to form a question you want to ask.

Toss the question and all the other balls into a tumbler and shake it all up for a while. Remove all the balls that didn't stick to the question.

What you have is not a "thought".

You have something that looks like a thought because it reflects actual thoughts that people did have, which all got encoded into the rules that made up the whole aparatus.

People created the alphabet and vocabulary written on the balls.

People created the associative meanings and encoded it into syntax and grammar rules, the weights, magnets, and shapes of the balls.

A person somewhere had a thought that there is a thing they will call the sky, and a sensation they will call blue, and an association that the sky is blue, and another association that "the sky is blue" is an assertion, and that another type of communication is a query, and that an assertion is a reasonable response to a query.

That is all represented in the construction of the balls. Out of all the purely random possible results, it's slightly more likely for the shake-up to produce "the sky is blue" because it fits a little better than other things against the seed crystal of your question.

This bingo tumbler produced a communication yet did not have a thought.

Most, maybe all? communication is some form of mechanistic encoding of thoughts. It's always possible to copy it or fake it, because it's not the consciousness itself, it's just something the consciousness caused to happen.

Some writing on a paper is not a thought, it's a picture of a thought.

The picture can be reproduced without the original thought occurring again. A new piece of paper can have a new instance of the writing spring forth without any conscious process behind it.

If you write something on a piece of paper, that was a person expressing a thought.

Now that piece of paper with writing on it lays on top of another peice of paper in the sun long enough for the sun to brown both papers. But the shadow from the ink transfers a duplicate inverse image onto the underlying paper that doesn't yellow as much.

That was a communication being reproduced. The written message on the 2nd paper did not exist, and then it did exist. What created it? Where did it come from? Is the first paper conscious and decided to communicate it's thoughts to you?

The first paper did not speak a thought via the 2nd paper, even though you can read the 2nd paper and interpret it as being the result of a coherent conscious thought. Neither the 1st nor 2nd pieces of paper thought anything. Merely ultimately a consciousness did cause the first paper to have an encoded representation of their thoughts on it, by writing them there.

That is the only reason the 2nd removed copy looks like a message. It is a message, but it's not a message from the piece of paper itself.

Even though the piece of paper is made out of complex carbon compounds "just like humans ZOMG!!!!!"