Comment by jukea

I was thinking the same. if there's anything that is what it is like to be an LLM (and I'm not saying that there is - in fact, I doubt it, while supposing that it is a possibility for future machines) I suspect it would be like this, but more so, and inverted: while Keller had some experience of an external world but no experience of language, the entire universe for an LLM is language, without any obvious way to suppose that this language is about an external world.

I think that LLMs might go through the reverse journey, being fluent in tokens (words-ish) and working backwards towards the physical reality we all inhabit.

i think the "problem" here is that for all of human history we have always been able to use mastery of language as a signal for intelligence and competence, of which LLMs are neither. it's possible this is even instinctual it's so ingrained in our concepts of "other minds". so we're going to have to get used to the fact that just using language well isn't enough to prove intelligence, certainly not consciousness.

which then begs the the question, what is the magic ingredient, on top of use of language, that we have that bestows these qualities?

and also the observation that whatever this ingredient is, it must be very difficult to measure or prove which is maybe why we stuck with the crude, but easy to wield, "use-of-language" test for so long.

Available to read online, I read it last year: https://rifters.com/real/Blindsight.htm

"We do not like annoying cousins." Yes, exactly. The, uh, confident fluency of LLM responses, which can at the same time contradict what was said earlier, reminded me exactly of that. I don't know if you've ever met one of those glib psychopaths, but they have this characteristic of non-content communication, where it feels like words are being arranged for you, like someone composing a song using words from a language they do not know. See also: "you're talking a lot, but you're not saying anything."

I think that this is false, as in intersubjectively not true for the human experience. First, because our physical state has a huge influence on our thoughts, not just their content, but direction, "color."

Secondly, and more importantly, while some thoughts may not have a language (image memories, mental maps), others certainly do, they're narrative. I only speak two languages but well enough (English is my second language) that I can think in both, and often come to a point where I have to decide which it will be for this train of thought.

Shape rotators vs wordcels distinction strikes again, I guess.

Do you have an inner monologue, out of curiosity? Because I absolutely think in English.

Based on the fact that people speaking different languages can lack basic abstract concepts or reason about them very differently, I think thoughts do have a language or at least often follow a language.

Here's a link to a transcript of a lecture with some very interesting examples: https://irl.umsl.edu/oer/13/

A quote as a sample: So let me tell you about some of my favorite examples. I'll start with an example from an Aboriginal community in Australia that I had the chance to work with. These are the Kuuk Thaayorre people. They live in Pormpuraaw at the very west edge of Cape York. What's cool about Kuuk Thaayorre is, in Kuuk Thaayorre, they don't use words like "left" and "right," and instead, everything is in cardinal directions: north, south, east and west. And when I say everything, I really mean everything. You would say something like, "Oh, there's an ant on your southwest leg." Or, "Move your cup to the north-northeast a little bit." In fact, the way that you say "hello" in Kuuk Thaayorre is you say, "Which way are you going?" And the answer should be, "North-northeast in the far distance. How about you?"

It's not growing, it's cutting and amplifying that happens on this step. Without it there is less certainty the attention will go the right way. When it is fully pronounced- as if you heard it with your ears - it sort of ripples through your brain, like a physical sensation, echoing for some time. Pulling it out word by word feels natural so You think it was already there, but The thought is not "activated" full strength until you walk it fully with attention, theres only a vague shape, a direction, broad and uncertain. Without it you get faster less lasting thoughts that won't chain well to form a cohesive picture. less sharpness, less resolution, a kind of intuition, maybe it is intuition. You probably get those sometimes, a complex idea that is hard to verbalise, but you can sense it, and you are probably aware how ephemeral they are if you dont stop to give them all your attention. You already know how to think like that, it just doesn't register that you are doing it because it doesn't "echo", you still walk them to remember them

I also find it astonishing that I can feel like I had an entire sentence in my head without any of the words, and fluidly produce all of the words as I say them, without having to search for them or consciously line them up. They're just there, one after the other, like tokens waiting to be picked up. (LLM anyone?) I don't even think that my conscious brain knows exactly which words are going to pop out, say 5 words on. It seems to magically find each word as I speak, without having to pause or rebuffer.

I don't think that language is slowing me down. I actually think that my brain is full of shit and needs to run thoughts through checkers (lint, syntax, logic, fact). I think it makes the language center of our brains all the more magical. As you say, it all happens so fast, and yet it assembles and sanity-checks those raw thoughts as you crystalise them into words.

How many times have I started explaining something, only to realise midway through that I'm taking crap, or that I'm extremely fuzzy on some important detail. Or maybe I infer some important new fact or make some new connection for the first time, while talking about it?

Dogs have thoughts... but we can speak. And every time there's been an innovation in the storage, retrieval or communication of language (not raw thoughts), we've had a gigantic evolutionary leap forward. Isaac Newton was a genius. But when he took up the challenge of explaining the motion of the planets, I bet that not even he knew what he was going to end up with at the end, and I bet that he realised, discovered or rained out a whole bunch of things in the writing of it.

Something else I've wondered. How come my brain holds a million different facts, records of! historical interactions with others, and a pretty decent track of time (like, I know the time, day, month and year and what I did-or-didn't do yesterday), but my dreams are total gibberish? Like I was in a hotel lobby last night with a bunch of people I don't know, realised I'm wasn't wearing any pants, then paniced because my phone was in my pants, how would I call my wife? So I turn to my (deceased) sister and asked which room I'm staying in... If my brain is so good, how come it does crap like that when the conscious bit is switched off?

I would never assume that the data inside my brain, or the subconscious babble that counts for thought, adds up to a genius that is hindered by some clunky language. Very much the opposite.

Side note: all of this is the basis for my extremely strong view that freedom of speech is an absolute necessity for continued prosperity, science, democracy etc. If people are unable to turn their ideas into concrete language, and to do this together as a group, without fear, then they are unable to reason things out properly and make good decisions. I only feel like adding that because within my lifetime I have seen an erosion of the importance of that freedom, to the point where it's no longer possible to discuss mundane, everyday things, or to point out some obvious truth.

Yes - to me the speech feels more like a forcing layer that drives thought. The actual thought is a recurrent neural network underneath. Nearest I have of conscious access to itis non verbal awareness of complex concepts.

That's interesting. I read your comment in my head while counting and seemed able to keep up, but something more complex might be hard, such as reading out loud.

On another tangent, I've been trying Ritalin after speaking to a doctor. The first thing I noticed when I took it was that it became very difficult to hold multiple trains of thought at the same time. My typical routine (coping mechanism) was to work and have YouTube playing and a little attention on each, because this stopped me getting bored. But it wasn't long before I realised I simply could not hear the videos. It was a strange feeling but nice. A little similar to what you describe in how abilities vary.

> This reminds me of some experiment (that I will never be able to find again)

That was from Richard Feynman.

I have been learning English for close to ~18 years by now, if you count primary school. To this day I can't really count in English unless I force myself to.

This is from Surely You’re Joking

The answer is that it is hugely variable between people!

Hurlburt has great research on this using Descriptive Experience Sampling.

Some people mainly use images, others mainly speech, others mainly emotion etc. And many more use a varied mix.

Also the way each modality of thought is used is hugely variable - exactly what people see and with what quality or how precisely they feel emotional in their body etc.

To me it explains a huge amount of how different people are good at different skills.

I've a podcast on this topic ("Imagine an apple") if you're interested in more.

You can visualize an algorithm?? Makes no sense to me. To me, when thinking about an algorithm, it's more navigating the data flow. Following connections between concepts. No words nor visuals.

It's not 100% for me but just the vast majority. I do visualise things that are almost purely spacial like geometry or recalling how to do an exercise. Though from what I've read, even this is news to some people who express surprise that "mind's eye" is a little more literal than they assumed. I'm pretty good at remembering facts and trivia but not so much actual life experiences, not sure if that's related.

I can visualize algorithms but I have to do so deliberately. Unlike the parent poster I don't always think in internal monologues either.

Sometimes it's a keyword/concept thing where I'll think of the main items and I get a feeling that I know how to fill in the blanks. I haven't actually visualized or verbalized what would fill those blanks though (and sometimes the feeling is wrong).

I think pretty much all of the senses can be used to do some form of thinking. I can imagine songs in my head, touch, feelings etc. Rarely are they useful for problem solving though, but some of these are nice for falling asleep in unknown environments.

Oh and then there's the thinking where nothing seems to happen. I stare at a piece of paper and after a while I know what to do next. How did I arrive at that conclusion? I don't know, but it definitely wasn't verbal, visual, aural or anything else. This tends to not solve complex problems like math, but it basically tells me what I should do to try to solve it (usually verbal or visual).