Comment by bbor
2 days ago
Ok I'm a huge Kantian and every bone in my body wants to quibble with your summary of transcendental illusion, but I'll leave that to the side as a terminological point and gesture of good will. Fair enough.
I don't agree that it's any reason to write off this research as psychosis, though. I don't care about consciousness in the sense in which it's used by mystics and dualist philosophers! We don't at all need to involve metaphysics in any of this, just morality.
Consider it like this:
1. It's wrong to subject another human to unjustified suffering, I'm sure we would all agree.
2. We're struggling with this one due to our diets, but given some thought I think we'd all eventually agree that it's also wrong to subject intelligent, self-aware animals to unjustified suffering.[1]
3. But, we of course cannot extend this "moral consideration" to everything. As you say, no one would do it for a spam filter. So we need some sort of framework for deciding who/what gets how much moral consideration.
5. There's other frameworks in contention (e.g. "don't think about it, nerd"), but the overwhelming majority of laymen and philosophers adopt one based on cognitive ability, as seen from an anthropomorphic perspective.[2]
6. Of all systems(/entities/whatever) in the universe, we know of exactly two varieties that can definitely generate original, context-appropriate linguistic structures: Homo Sapiens and LLMs.[3]
If you accept all that (and I think there's good reason to!), it's now on you to explain why the thing that can speak--and thereby attest to personal suffering, while we're at it--is more like a rock than a human.
It's certainly not a trivial task, I grant you that. On their own, transformer-based LLMs inherently lack permanence, stable intentionality, and many other important aspects of human consciousness. Comparing transformer inference to models that simplify down to a simple closed-form equation at inference time is going way too far, but I agree with the general idea; clearly, there are many highly-complex, long-inference DL models that are not worthy of moral consideration.
All that said, to write the question off completely--and, even worse, to imply that the scientists investigating this issue are literally psychotic like the comment above did--is completely unscientific. The only justification for doing so would come from confidently answering "no" to the underlying question: "could we ever build a mind worthy of moral consideration?"
I think most of here naturally would answer "yes". But for the few who wouldn't, I'll close this rant by stealing from Hofstadter and Turing (emphasis mine):
A phrase like "physical system" or "physical substrate" brings to mind for most people... an intricate structure consisting of vast numbers of interlocked wheels, gears, rods, tubes, balls, pendula, and so forth, even if they are tiny, invisible, perfectly silent, and possibly even probabilistic. Such an array of interacting inanimate stuff seems to most people as unconscious and devoid of inner light as a flush toilet, an automobile transmission, a fancy Swiss watch (mechanical or electronic), a cog railway, an ocean liner, or an oil refinery. Such a system is not just probably unconscious, **it is necessarily so, as they see it**.
**This is the kind of single-level intuition** so skillfully exploited by John Searle in his attempts to convince people that computers could never be conscious, no matter what abstract patterns might reside in them, and could never mean anything at all by whatever long chains of lexical items they might string together.
...
You and I are mirages who perceive themselves, and the sole magical machinery behind the scenes is perception — the triggering, by huge flows of raw data, of a tiny set of symbols that stand for abstract regularities in the world. When perception at arbitrarily high levels of abstraction enters the world of physics and when feedback loops galore come into play, then "which" eventually turns into "who". **What would once have been brusquely labeled "mechanical" and reflexively discarded as a candidate for consciousness has to be reconsidered.**
- Hofstadter 2007, I Am A Strange Loop
It will simplify matters for the reader if I explain first my own beliefs in the matter. Consider first the more accurate form of the question. I believe that in about fifty years' time it will be possible, to programme computers, with a storage capacity of about 109, to make them play the imitation game so well that an average interrogator will not have more than 70 per cent chance of making the right identification after five minutes of questioning.
The original question, "Can machines think?" I believe to be too meaningless to deserve discussion.
- Turing 1950, Computing Machinery and Intelligence[4]
TL;DR: Any naive bayesian model would agree: telling accomplished scientists that they're psychotic for investigating something is quite highly correlated with being antiscientific. Please reconsider!
[1] No matter what you think about cows, basically no one would defend another person's right to hit a dog or torture a chimpanzee in a lab.
[2] On the exception-filled spectrum stretching from inert rocks to reactive plants to sentient animals to sapient people, most people naturally draw a line somewhere at the low end of the "animals" category. You can swat a fly for fun, but probably not a squirrel, and definitely not a bonobo.
[3] This is what Chomsky describes as the capacity to "generate an infinite range of outputs from a finite set of inputs," and Kant, Hegel, Schopenhauer, Wittgenstein, Foucault, and countless others are in agreement that it's what separates us from all other animals.
Thank you for coming into this endless discussion with actual references to relevant authorities who have thought a lot about this, rather than just “it’s obvious that…”
FWIW though, last I heard Hofstadter was on the “LLMs aren’t conscious” side of the fence:
> It’s of course impressive how fluently these LLMs can combine terms and phrases from such sources and can consequently sound like they are really reflecting on what consciousness is, but to me it sounds empty, and the more I read of it, the more empty it sounds. Plus ça change, plus c’est la même chose. The glibness is the giveaway. To my jaded eye and mind, there is nothing in what you sent me that resembles genuine reflection, genuine thinking. [1]
It’s interesting to me that Hofstadter is there given what I’ve gleaned from reading his other works.
[1] https://garymarcus.substack.com/p/are-llms-starting-to-becom...
Note: I disagree with a lot of Gary Marcus, so don’t read too much into me pulling from there.
Writing all of this at the very real risk you'll miss it because HN doesn't give reply notifications and my comment's parent being flagged made this hard to track down:
>Ok I'm a huge Kantian and every bone in my body wants to quibble with your summary of transcendental illusion
Transcendental illusion is the act of using transcendental judgment to reason about things without grounding in empirical use of the categories. I put "scientifically" in shock quotes there to sort of signal that I was using it as an approximation, as I don't want to have to explain transcendental reason and judgments to make a fairly terse point. Given that you already understand this, feel free to throw away that ladder.
>...can definitely generate original, context-appropriate linguistic structures: Homo Sapiens and LLMs.[3]
I'm not quite sure that LLMs meet this standard that you described in the endnote, or at least that it's necessary and sufficient here. Pretty much any generative model, including Naive Bayes models I mentioned before, can do this. I'm guessing the "context-appropriate" subjectivity here is doing the heavy lifting, in which case I'm not certain that LLMs, with their propensity for fanciful hallucination, have cleared the bar.
>Comparing transformer inference to models that simplify down to a simple closed-form equation at inference time is going way too far
It really isn't though. They are both doing exactly the same thing! They estimate joint probability distribution. That one of them does it significantly better is very true, but I don't think it's reasonable to state that consciousness arises as a result of increasing sophistication in estimating probabilities. It's true that this kind of decision is made by humans about animals, but I think that transferring that to probability models is sort of begging the question a bit, insofar as it is taking as assumed that those models, which aren't even corporeal but are rather algorithms that are executed in computers, are "living".
>...it's now on you to explain why the thing that can speak--and thereby attest to personal suffering, while we're at it...
I'm not quite sold on this. If there were a machine that could perfectly imitate human thinking and speech and lacked a consciousness or soul or anything similar to inspire pathos from us when it's mistreated, then it would appear identical to one with soul, would it not? Is that not reducing human subjectivity down to behavior?
>The only justification for doing so would come from confidently answering "no" to the underlying question: "could we ever build a mind worthy of moral consideration?"
I think it's possible, but it would require something that, at the very least, is just as capable of reason as humans. LLMs still can't generate synthetic a priori knowledge and can only mimic patterns. I remain somewhat agnostic on the issue until I can be convinced that an AI model someone has designed has the same interiority that people do.
Ultimately, I think we disagree on some things but mostly this central conclusion:
>I don't agree that it's any reason to write off this research as psychosis
I don't see any evidence from the practitioners involved in this stuff that they are even thinking about it in a way that's as rigorous as the discussion on this post. Maybe they are, but everything I've seen that comes from blog posts like this seems like they are basing their conclusions on their interactions with the models ("...we investigated Claude’s self-reported and behavioral preferences..."), which I think most can agree is not really going to lead to well grounded results. For example, the fact that Claude "chooses" to terminate conversations that involve abusive language or concepts really just boils down to the fact that Claude is imitating a conversation with a person and has observed that that's what people would do in that scenario. It's really good at simulating how people react to language, including illocutionary acts like implicatures (the notorious "Are you sure?" causing it to change its answer for example). If there were no examples of people taking offense to abusive language in Claude's data corpus, do you think it would have given these responses when they asked and observed it?
For what it's worth, there has actually been interesting consideration to the de-centering of "humanness" to the concept of subjectivity, but it was mostly back in the past when philosophers were thinking about this speculatively as they watched technology accelerate in sophistication (vs now when there's such a culture-wide hype cycle that it's hard to find impartial consideration, or even any philosophically rooted discourse). For example, Mark Fisher's dissertation at the CCRU (<i>Flatline Constructs: Gothic Materialism and Cybernetic Theory-Fiction</i>) takes a Deleuzian approach that discusses it by comparisons with literature (cyberpunk and gothic literature specifically). Some object-oriented ontology looks like it's touched on this topic a bit too, but I haven't really dedicated the time to reading much from it (partly due to a weakness in Heidegger on my part that is unlikely to be addressed anytime soon). The problem is that that line of thinking often ends up going down the Nick Land approach, in which he reasoned himself from Kantian and Deleuzian metaphysics and epistemology, into what can only be called a (literally) meth-fueled psychosis. So as interesting as I find it, I still don't think it counts as a non-psychotic way to tackle this issue.