Comment by sirwhinesalot

I generally agree about the problem with anthropomorphizing. But I don't think Anthropic are doing that. They explicitly write "in biological entities this would be considered a sign of consciousness, but we don't know how to interpret it here".

However, I disagree with your point that "it's an autoregressive function, thus it doesn't matter". Let me explain why:

Assume I do a complete neurological scan of a brain. I then implement this scan in a simulation and run it. Assume that my scan and my simulation of the biology of the brain (and the sensory and motorical inputs and outputs) is good enough that you can now have conversations with the simulation, and in all aspects, this simulation behaves exactly like you expect a human to behave.

Of course this is deterministic. If you take the state of the brain and then run it again, replaying the inputs, you get the exactly same behavior again.

I would argue that the experiences of this simulation are of the same onthological status as our own.

Now I work in dynamical systems. The autoregressive process of LLMs (hooked up to a harness providing it with inputs and outputs) is roughly in the same complexity class I would expect for a brain simulation. A physical simulation of an ODE is also an autoregressive process. The major major difference here is the existence of a latent brain state. But conversely the autoregression on sequences of hundreds of thousands of tokens is a much higher dimensional state than I expect for the latent brain state. In my view this is more an artifact of our inefficient LLM architectures, than a fundamental difference.

Now to be absolutely clear: I don't see evidence that would clearly suggest that LLMs have experiences on the same onthological status as we do. I simply believe this is a reasonable and relevant question to ask.

> Not the same person but to me, the answer is that it does not matter, and that all these attempts at making it matter are pure marketing and emotional manipulation.

This is an opinion that has no basis in any meaningful conceptual framework other than I am human and I want to feel special about it.

> It's not a living creature.

You mean, it is not biological life. And sure, that is the default meaning of life. We soon may have to extend it to digital life as well, or we will have to consider "conscious digital exitance" as a life analogue. At any rate, it has never been seriously argued that consciousness requires a biological substrate, see the thought experiments regarding computer simulations of the human brain. Would that not be a function as well, completely predictable because it is "just a program"? If not, then why not? And how does that differ from the predictability or reproducibility of LLM outputs?

My point is, all current proof points in a direction that strongly suggests that you need to reevaluate your first principles on this topic.

Humans are not living creatures. They're just bipedal meat shells being operated by a 20W electrochemical computer running a suite of chemically signalled, electrically actuated modellable functions, much of which is wasted on homeostatic regulation of the meat shell, which is capable of incredible things, but it's still just a result of simple electrochemical functions like action potential generation, dendritic integration, AMPA NMDA GABA receptor dynamics, attractor memory, excitation/inhibition balance, PING/ING gamma oscillations, basal ganglia action selection, hippocampal coding, astrocyte calcium signaling, etc. It is not alive as it cannot conform to my preferred arbitrary priors about aliveness, like being able to rapidly divide 30 digit integers the way truly intelligent beings can. It is less "alive" than the TI-83 your mother bought you for your high school math classes. If it simulates something resembling consciousness that's neat but no more relevant than a more complex version of Conway's game of life.

Jokes aside, the map is not the terrain. We can enumerate the understood first-order electrochemical mechanisms in the human brain in the same way we can enumerate the understood first-order sampling and token prediction mechanisms in an LLM. Nobody serious in neuroscience will tell you that we exhaustively understand every single aspect of human cognition and the human brain, just as nobody serious in AI/ML will tell you that we exhaustively understand every single aspect of LLM "cognition" and the latent space networks that LLMs use internally. Our map of how each of these complex systems work is a simplified enumeration of the components we do understand, not an exhaustive and perfectly accurate enumeration of how they actually work.

This is why there is a steady stream of research being churned out discovering complex emergent properties in LLMs and their latent spaces. If you're not aware of it already, Anthropic's research on "J-Space" is a fascinsting look into an apparent observed emergent mechanism within an LLMs internal activations closely resembling global workspace theory in human cognition.

Nobody deliberately designed this "global workspace", it was an emergent property in a sufficiently complex system that we had limited visibility and insight into.

Seemingly simple systems have these emergent complex properties all over the place. Conway's game of life is about as simple of a set of rules as you can get, yet has all sorts of complex emergent behaviors like gliders, oscillators, LWSS/MWSS/HWSS, guns, puffers, rakes, reflectors, logic gates, and even whole turing machines. Nobody programmed a single one of these complex patterns in, they emerged from a simple set of rules.

To be clear, I'm not making the argument that LLMs definitely are conscious, I'm making the argument that we don't understand enough about them to assert with absolute confidence that they aren't. Human history is rife with a long list of consciousness being denied to "the other" - different ethnicities, different genders, differently abled, even different species. The side of "They're not conscious" has a lengthy track record of being wrong over and over again. Why not have just a sliver of intellectual humility about what we don't know?

As an aside to my main point - Also, what's with the handwringing over people ERPing with an LLM? Is it mental illness when people sincerely believe in astrology, or tarot cards, or voodoo, or organized religion that says the earth is 6000 years old? Most humans believe silly, unempirical things. What about when they watch adult video in VR, or have waifus? Humans engage in voluntary suspension of disbelief for pleasure and recreation all the time. As long as they're not infringing upon the rights of anyone else, what's the big deal? Who put you in charge as the head of the belief police?