Comment by jlhawn
15 hours ago
Now I can't stop thinking about _The Experience Machine_ by Andy Clark. It theorizes that this is how humans navigate and experience the real world: Our brains generate what we think the world around is like and our senses don't so much directly process visual information but instead act like a kind of loss function for our internal simulations. Then we use that error to update our internal model of the world.
In this view, we are essentially living inside a high-fidelity generative model. Our brains are constantly 'hallucinating' a predicted reality based on past experience and current goals. The data from our senses isn't the source of the image; it's the error signal used to calibrate that internal model. Much like Genie 3 uses latent actions and frames to predict the next state of a world, our brains use 'Active Inference' to minimize the gap between what we expect and what we experience.
It suggests that our sense of 'reality' isn't a direct recording of the world, but a highly optimized, interactive simulation that is continuously 'regularized' by the photons hitting our retinas.
This is one of my fundamental beliefs about the nature of consciousness.
We are never able to interact with the physical world directly, we first perceive it and then interpret those perceptions. More often than not, our interpretation ignores and modifies those perceptions, so we really are just living in a world created by our own mental chatter.
This is one of the core tenets of Buddhism, and it's also expounded on Greg Egan's short novel "Learning to Be Me". He's one of my favorite sci-fi authors and this particular short led me down a deep rabbit hole of reading many of his works within a few months.
I found a copy online, if you haven't read it, do yourself a favor and check it out. You won't be able to put it down and the ending is sublime. https://gwern.net/doc/fiction/science-fiction/1995-egan.pdf
“ This is one of my fundamental beliefs about the nature of consciousness. We are never able to interact with the physical world directly, we first perceive it and then interpret those perceptions. More often than not, our interpretation ignores and modifies those perceptions, so we really are just living in a world created by our own mental chatter.”
This is an orthodox position in modern philosophy, dating back to at least Locke, strengthened by Kant and Schopenhauer. It’s held up to scrutiny for the past ~400 years.
But really it’s there in Plato too, so 2300+ years. And maybe further back
It’s the Allegory of the Cave, isn’t it?
1 reply →
This is absolutely what happens. It's even more tricky since our sensory inputs have different latencies which the brain must compile back into something consistent. While doing so it interprets and filters out a lot of unsurprising, expected data.
https://www.youtube.com/watch?v=wo_e0EvEZn8
Thank you for linking this! I'm a big fan of Egan but had never read this particular short story. I feel like Egan is perhaps the only contemporary author who actually _gets_ consciousness.
I think this is pretty well established as far as neurologists are concerned and explains a lot of things. Like dreaming for instance.. just something like the model running without sensory input constraining it.
Always wondered if dreaming is some kind of daily memory consolidation function. Logged short-term/episodic memory being filtered and the important bits baked by replaying in a limited simulacrum.
There was once a neural network that used dreaming phases for regularisation. It would run in reverse on random data and whatever activated was down–weighted.
2 replies →
Could you please give some sources - books or articles or videos on that topic? It's really fascinating
https://onlinelibrary.wiley.com/doi/abs/10.1111/phib.12268?u...
https://pubmed.ncbi.nlm.nih.gov/23663408/
https://royalsocietypublishing.org/rstb/article/371/1708/201...
https://pubmed.ncbi.nlm.nih.gov/20068583/
I'll also recommend Being You by Seth Anil. It makes a lot of sense of consciousness to me. It certainly doesn't answer the question but it's not just throw your hands up and "we have no idea why qualia", and it's also not just "here's a list of neural correlates of consciousness and we won't even discuss qualia".
It goes through how sensations fit into this highly constrained, highly functional hallucination that models the outside world as a sort of bayesian prediction about the world as they relate to your concerns and capabilities as a human, and then it has a very interesting discussion about emotions as they relate to inner bodily sensations.
the book I mentioned (_The Experience Machine_ by Andy Clark) talks about this.
https://slatestarcodex.com/2017/09/05/book-review-surfing-un...
A kurzgesagt on this: Why Your Brain Blinds You For 2 Hours Every Day https://youtu.be/wo_e0EvEZn8 and the sources for that video - https://sites.google.com/view/sources-reality-is-not-real/
Like, "Your Brain Hallucinates Your Conscious Reality" as exposed by Anil Seth[1]? Found that one while searching for something like "the illusion of the self" a few years ago.
It’s also easy to find this treated in various philosophy/religion through time and space. And anyway as consciousness is eager to project whatever looks like a possible fit, elements of suggesting prior arts can be inferred back as far as traces can be found.
[1] https://www.youtube.com/watch?v=lyu7v7nWzfo
Doesn't this have some implications for P vs NP?
How much compute do you need to convince a brain its environment is "real"?
What happens if I build a self replicating super computer in this environment that finds solutions to some really big SAT instances that I can verify?
Dreams run into contradictions quite quickly.
Also check out The Case Against Reality by Donald Hoffman
Yes!
Also See essentia foundation videos
https://youtube.com/@essentiafoundation?si=aD-RmB8DF4M_Oc7w
It's kinda obvious if you think about this:
- How come we have 2 eyes but see one 3d world?
- We hear sounds and music coming from various directions, but all of this is created from 2 vibrating eardrums
Another analogy that kinda fits in with what you're saying is the post-processing on smartphone "photos."
At what point does the processing become so strong that it's less a photograph and more a work of computational impressionism?
At the point where Samsung detects a photo of a white circle while the phone is pointing upwards and substitutes a high resolution picture of the moon.
This actually happened.
This is easily corroborated by taking hallucinogens. Your subjective experience is a simulation, augmented by your senses.
Personally I often catch myself making reading mistakes and knowing for a fact that the mistake wasn't just conceptual, but an actual visual error where my brain renders the wrong word. Sometimes it's very obvious because the effect will last for seconds before my vision "snaps" back into reality and the word/phrase changes.
I first noticed this phenomenon in my subjective experience whenever I was 5 and started playing Pokémon. For many months, I thought Geodude was spelled and pronounced Gordude, until my neighbor said the name correctly one day and it "unlocked" my brain's ability to see the word spelled correctly.
The effect is so strong sometimes that I can close my eyes and imagine a few different moments in my life, even as a child, where my brain suddenly "saw" the right word while reading and it changed before my eyes.
Just want to say this is a really good description of our brain's simulation, and I have experienced the same catching-the-misread-word phenomenon, and it's a subtle reminder about how this is all working. But does this mean our wires are crossed in a particular way that is uncommon? I haven't heard others share a similar experience.
I'm not sure. At times I've wondered if I have something similar to dyslexia. There are few common failure modes with me such as flipping consonants or vowels between adjacent words, or writing down a word and it being the wrong one.
My brain seems to store/recall words phonetically, possibly because I taught myself to read at age 3 with my own phonetic approach, but also possibly due to how I trained myself out of a long spell of aphasia during high school by consciously relearning how to speak in a way that engaged the opposite hemisphere of my brain; thinking in pitches, intonation, rhyme, rhythm, etc. and turning speaking into a musical expression. I'd read about this technique and after months of work I managed to make it work for me. So in that aspect, there really might be some crossed wires out of necessity.
I was homeless in high school and thus too poor to visit doctors and get scans done, so I'm really not sure if the assumed damage to my left hemisphere which I experienced was temporary or permanent, or even detectable. The aphasia was coupled with years of intense depersonalization and derealization as well. The brain is a very strange thing and many events in my life such as the ones described above have only reinforced to me how subjective my experience really is.
Yeah, this kind of thing was part of the subject of my PhD, first postdoc, and ongoing scientific work. The question is how to produce generative models and inverse-inference algorithms that are powerful enough to work in tens to hundreds of milliseconds in high dimensionality :-/
Let me introduce you to Idealism
And more specifically Analytic Idealism
https://youtu.be/P-rXm7Uk9Ys?si=q7Kefl7PbYfGiChZ
Google DeepMind’s Project Genie is being framed as a “world model.” Given a text prompt, it generates a coherent, navigable, photorealistic world in real time. An agent can move through it, act within it, and the world responds consistently. Past interactions are remembered. Physics holds. Cause and effect persist.
From a technical standpoint, this is impressive engineering. From a philosophical standpoint, it’s an unexpectedly clean metaphor.
In analytic idealism, the claim is not that the physical world is fake or arbitrary. The claim is that what we call the “physical world” is how reality appears from a particular perspective. Experience is primary. The world is structured appearance.
Genie makes this intuitive.
There is no “world” inside Genie in the classical sense. There is no pre-existing ocean, mountain, fox, or library. There is a generative substrate that produces a coherent environment only when a perspective is instantiated. The world exists as something navigable because there is a point of view moving through it.
Change the character, and the same environment becomes a different lived reality. Change the prompt, and an entirely different universe appears. The underlying system remains, but the experienced world is perspective-dependent.
This mirrors a core idealist intuition: reality is not a collection of objects waiting to be perceived. It is a structured field of possible experiences, disclosed through perspectives.
The interesting part is not that Genie “creates worlds.” It’s that the worlds only exist as worlds for an agent. Without a perspective, there is no up, down, motion, danger, beauty, or meaning. Just latent structure.
Seen this way, Genie is not a model of consciousness. It’s a model of how worlds arise from viewpoints.
If you replace “agent” with “local mind,” and “world model” with “cosmic mental process,” the analogy becomes hard to ignore. A universal consciousness need not experience everything at once. It can explore itself through constrained perspectives, each generating a coherent, law-bound world from the inside.
That doesn’t prove idealism. But it makes the idea less mystical and more concrete. We are already building systems where worlds are not fundamental, but perspectival.
And that alone is worth sitting with.
It's pretty clear you used an LLM to write that, given your post history. I'm not sure that's allowed here, but at least put a disclaimer.
Yes I used an LLM - to post my thoughts from my phone as typing that down and spell checking/grammar cleanup is hell on mobile -
- but does it mean I don’t believe all the words written above are valid? No absolutely not.
I reviewed and copyedited what I posted and the meaning is exactly what I intended to post so I’m not sure what’s the issue here
If we use LLMs to expound on our own thoughts is it a crime? They are literal masters of wordplay and rote clarification on complex topics so I think this is a very legitimate use-case for them, since I was going for clarity as an objective- esp considering the topic
Also none of my previous posts were LLM written (including this one)
People are a little over-sensitive on this topic these days
Consciousness and perspective are temporally stable fixed points in the universe. You come to understand yourself as "you" or "I" because it's the only thing in the world around you that does not immediately change under many transformations.
For example, you can spin around, or change position, or close your eyes, and you're still you. As you navigate and interact with the evolving universe, the only continual, relatively unchanging part of the experience is what your brain uses to differentiate itself from the rest of your perceptions.