Comment by cauch

> The point is that as a solution to certain kinds of optimization problems, consciousness can conceivably be the solution to the optimization problem of predicting the next token of text written by humans who themselves have complex phenomenology.

Yes, I agree with that. Consciousness is a good way of generating convincing human text.

What I don't agree with is that consciousness is the only way to generate convincing human text and that because we have convincing human text, it can only imply we have consciousness.

There is a huge probability that generating convincing human text can be done without consciousness. Either because there are efficient mechanisms as efficient as the way the human brain deal with this problem and that the LLM found one of them (and these mechanism may be quite difficult to imagine for a human). Or even because the LLM found a local minimum and is stuck there.

To re-use the evolution approach: evolution solved the "flying problem" with bird feathers, but also with insect wings or bat wings. The fact that evolution ended up using feather does not imply that everything that flies can only fly with feathers.

> World models don't have to be rich and detailed to count as a world model

I agree in general, but here, we are talking about machine that reproduce all human language. The argument I'm answering to is pretending that "all of human knowledge" is understood, which include every single human concept. This has to be everything, because LLM is able to provide convincing text about every subject. If on some subject, the LLM is able to provide convincing text without "understanding" it, then the argument that it is impossible to provide convincing text without understanding it collapse.