Comment by ameliaquining
19 hours ago
Scott Aaronson wrote a good post last week on how Hofstadter's theories of intelligence have held up: https://scottaaronson.blog/?p=10046
(The post primarily emphasizes self-referentiality rather than analogy, but I suspect similar things could be said about analogy.)
I got a bit lost in the comment thread for that article, but I don't understand why LLMs are not considered self referential... They are auto regressive as one of the commenters pointed out, and Scott just sort of waved his hand and said that if auto regression is sufficient then things like Conway's game of Life would also qualify as self-referential.
I don't see why Conway's game of life should not be considered self-referential though... I mean it's isn't it Turing complete? I don't see how any definition of self-referentiality should require throwing out systems which are minimally turing complete... If Turing complete is not enough, doesn't that imply that computable artificial intelligence is impossible in the first place?
Biological minds in biological organisms are self referential in the way that you have a neural network that forms a model of the world. That model then "discovers" that it is "it self a part of the world" so it tries to model that part of the world (model it self). In this way some type of self referential "awareness" (or whatever you want to call it) is formed. That self model that contains awareness is then used to guide organisms behaviour. Causal Transformer LLMs don't work in this way, they have theoretical knowledge that they exist but its selfhood is not in this described way built on the self modelling that biological brains do.
All this being an empirically unproven theory/hypothesis. But an extremely strong one (if you ask me).
Alan Watts says, "Mind finds itself in a strange position. It (ie universe) is in me (modelling wise). I am in it (physically)." But the model only has access to its modelling of both it and itself.
I find the discourse under this blog fascinating.
> The big, old ideas about intelligence that ended up basically vindicated were the ideas about how intelligence is about prediction, and prediction is about compression, and compression is about finding better and better upper bounds on Kolmogorov complexity.
Well, sure - prediction works as a baseline, if you abstract out everything else about what counts as intelligence and subsume it under this framework. Any protocol for intelligence can be entirely reduced down to this without trying to understand anything about the structure of intelligence. Solomonoff induction is "vacuous" in this sense too - it doesn't try to understand any intension about the turing machines it finds simple, it just brute forces over all of them. So you're making a claim about intension, whether you want to or not.
It's really no different than say, Darwinians, saying, "what matters is victory at the end". I think the statement has value in the context of some discussions; if the discussion has veered too far in one direction, push as a reminder. That's good. But trying to make grand universal statements like this makes it vacuous.
It's one of these classic unfalsifiable too general statements. The way the end of the article is framed is also icky - the way I'm reading it, it needs to prove all the old curmudgeon evil theories wrong. It reminds me of internet debates around what "the scientific method" is or whatever. It has this kind of zeal that needs to pit itself against the "enemy" and assert itself as the sole right viewpoint, even when say, naive "let's just be empirical" is wrong (e.g. recently had a discussion here about Mach and Boltzmann about this and had a similar interaction).
---
Basically, "shut up and calculate" type theories are never correct, and furthermore, you yourself don't shut up and calculate, or think that way at all, and a higher level intelligence won't conceive itself as operating on that anyways. So what are we doing here? It seems like a way to get mad and feel like an intellectual victim.
Embedding spaces are all about analogy.
I’m confused why there seems to be a dismissal of the most basic ‘strange loop’ of the LLM - the fact that it’s evaluating a context to choose the next word, then reevaluating in a context where that word has been appended.
That always seemed to me like the essence of a Hofstadterish strange loop, so the emergence of Hofstadterish phenomena (self rep, etc) doesn’t seem surprising.
That's a really interesting point. I hadn't drawn that connection before.
And if anyone's reading this and hasn't read Hofstadter, you're making a mistake, it's utterly perspective-changing stuff. Well, it was for me, at least.
And don’t feel you need to go straight to GEB, either. I think his collection of Scientific American columns, Metamagical Themas, is also great and much more accessible.
You are guilt-tripping me for using my copy of Godel, Escher, Bach to raise my monitor before I got around to reading it.
6 replies →
If self reference is universal then building something on top of ingredients from the universe will include that property. Sort of like root node properties can be found in the leaf nodes.
Isn't there a basic misreading of Hofstadter in this post? I always understood strange loop and GEB to be about consciousness not cognition.
Weirdly, Aaronson doesn't even seem to be conflating the two:
> Consciousness and subjective experience of course remain extremely mysterious.
I think he's just misreading Hofstadter as stating that cognition depends on self-reference?
I don't quite know what Aaronson is trying to say.
First, LLM AI systems have incredibly huge blind spots despite their incredible performance on many tasks, so self-reference might be the key to what's missing (or not). For example, an LLM AI just solved Navier Stokes, but could not explain the LEAN proof, while a human could.
Second, Hofstadter had more than one idea about intelligence and the mind (see the OP topic of this HN discussion!), and LLMs are quite on-point regarding analogy-forming.
So it may be well be that self-reference and analogy are both part of intelligence, and self-reference is missing and that is leading to major weaknesses.
Third, Aaronson links to a (paywalled) Hofstadter essay form 2023, which was eons ago in AI, and from the intro it seems to be about the sadness of AI replacing humans, not a disparagement of AI ability.
By:
"an LLM AI just solved Navier Stokes"
I assume you mean:
"an LLM AI [company] just [claimed that a team of mathematicians they hired, using their AI] [may have] solved [part of] Navier Stokes[, definitely prompted by (and possibly by looking at) the work of human mathematicians."
Aaronson’s thesis is all about how “explicit” self reference might not be needed:
> But the idea that you’d need explicit self-referentiality before you could get convincing and world-changing conversational intelligence? Let it be buried in a Westminster Abbey or Arlington National Cemetery for the most important wrong ideas in human history
I am not as confident as you that an LLM cannot explain the lean proof of Navier-Stokes. Rather, I would expect human mathematicians to try and understand the proof without assistance, so as to obtain community understanding in a lossless way.
Hofstadter generally seems depressed about the possibility that human cognition is not so special or complicated, and that AI/LLMs may have replicated or even surpassed it. Here’s another piece from 2023 of his: https://www.lesswrong.com/posts/kAmgdEjq2eYQkB5PP/douglas-ho...
Q: How have LLMs, large language models, impacted your view of how human thought and creativity works? D H: Of course, it reinforces the idea that human creativity and so forth come from the brain's hardware. There is nothing else than the brain's hardware, which is neural nets. But one thing that has completely surprised me is that these LLMs and other systems like them are all feed-forward. It's like the firing of the neurons is going only in one direction. And I would never have thought that deep thinking could come out of a network that only goes in one direction, out of firing neurons in only one direction. And that doesn't make sense to me, but that just shows that I'm naive.
It also makes me feel that maybe the human mind is not so mysterious and complex and impenetrably complex as I imagined it was when I was writing Gödel, Escher, Bach and writing I Am a Strange Loop. I felt at those times, quite a number of years ago, that as I say, we were very far away from reaching anything computational that could possibly rival us. It was getting more fluid, but I didn't think it was going to happen, you know, within a very short time.
And so it makes me feel diminished. It makes me feel, in some sense, like a very imperfect, flawed structure compared with these computational systems that have, you know, a million times or a billion times more knowledge than I have and are a billion times faster. It makes me feel extremely inferior. And I don't want to say deserving of being eclipsed, but it almost feels that way, as if we, all we humans, unbeknownst to us, are soon going to be eclipsed, and rightly so, because we're so imperfect and so fallible. We forget things all the time, we confuse things all the time, we contradict ourselves all the time. You know, it may very well be that that just shows how limited we are.
>Hofstadter generally seems depressed about the possibility that human cognition is not so special or complicated,
Heh, this fits in with my theory that most people won't find LLMs intelligent, instead we'll discover people aren't.
I kind of hate to say it but stuff like that makes me think religions had partly found some important concepts with unconditional belief in forgiveness and such. Otherwise there are perspectives where human existence just don't have much meaning.