Comment by TeMPOraL

> Have you ever had a professor or mentor walk you through some experiences as a way to help you understand some complex concepts? I guess language is used in that but the ideas are conveyed by the process not the language.

Yes, but I believe this is happening with LLMs too. Language (as in text, symbols, actions) is a vehicle, but much like us, language models have internal models and represent concepts (this has been directly, empirically demonstrated few years ago), and they don't "think" in tokens either[0].

> And modalities that don’t include “where is this body in space” or “I am lonely” don’t seem to capture some essential elements of sensory experience. It is a mistake to separate the interior from the exterior in the analysis of how our thinking works.

That's fair. LLMs don't capture every dimension we experience. There's also history - our individual lived experiences since birth are, in my view, something else entirely. It's not a modality, but it's also not something currently possible to capture in or post training.

> I think the folks saying these systems have a kind of intelligence, but a non-human kind are correct - whether that can lead to an independent intelligence that can stay stable sane and focused for weeks or months as many people can, without some sort of embodied cognition providing over all wellness checks to keep the system of thinking sane, remains to be seen.

Possibly. I definitely agree it's not human intelligence. I think it's human-like, in the sense of human-approximating, by virtue of how it's trained[1], but it's arriving there via a different path so end result can still be alien (though I speculate approximation will hold[2]).

> even as just throwing all the ops data at them and asking what’s up is, several years in, still not worth doing.

I guess depends on the complexity of the case (and my understanding of your example); e.g. in my case, I had stellar results from giving Sonnet and Opus (from 4.6 all the way to now) access to my Home Assistant instance. They aren't perfect at making dashboards, but they're excellent at surfacing insights I didn't even realize were possible to get.

--

[0] - That's distinct from the "tokens are units of thinking" heuristic, which still holds for mechanistic reasons - best analogy IMO is "clock signal" in ICs.

[1] - The overall goal function is literally just "generate output that looks sensible to a human", in fully general, unqualified sense. Or, put another way, we're just brute-forcing DWIM, and rating the output by whether it's "what I meant".

[2] - Thinking about constraints on biological evolution, whatever the design of a human mind is, the fundamentals behind it must be so simple, that a greedy incremental optimizer random-walked into it. Given how far we've got with LLMs using simple architecture and crude training methods, and especially how eerily similar their failure modes are to human cognitive failure modes, I suspect LLMs are actually attracted towards the same fundamental design as evolution discovered.