Comment by antihipocrat
2 days ago
At the end of the page is this paragraph:
Another, more subtle justification for the Butlerian Jihad is also found in Frank Herbert's original novels, specifically Heidegger's thesis that the use of technology trains humans to think like machines. The problem is that machines are deterministic; thus, training people to be machines is self-limiting.
I wonder whether Heidegger would consider the probabilistic nature of LLMs as having a positive effect on human thought.
Although LLMs are not deterministic in normal operation (because of the temperature values used) they are still strongly biased in their outputs. Bias in their outputs is literally how they function.
So you get a different type of conformism where people who interact with LLMs a lot are going to be influenced towards the median, as encoded in those token probability look up tables.
The effect is basically the same as what Heidegger and Illich were worried about, and I think we're seeing it happen already everywhere.
A solution(?) is to try to be your true weird authentic self when you act in the world, rather than bending your words and behaviours to the machine and LLM probability shape.
I would like to state here, that LLMs are actually entirely deterministic. The temperature is just as much of an input as any other parameter and instructions and prompts. Deterministic does not mean "explainable" or "reasonable" or "understandable". Unless LLMs somehow have broken out of the confines of our very deterministic computers/calculation machines without us noticing, they are very much as deterministic as anything else we run on these machines.
What people perhaps mean by non-deterministic on a computer is, that we often do not control all inputs and therefore cannot tell what the output will be. That's however actually more about our own inability, rather than the processes within the machines.
If the temperature dependency is implemented via a deterministic pseudo-random algorithm, then they are deterministic. If, on the other hand, the output is at least partially causally dependent on a source of environmental randomness, then, even though everything 'inside the box' is deterministic, the process as a whole may or may not be, depending on whether the source of environmental randomness is, and that is ultimately a question about fundamental reality.
Aren't we also biased in our outputs? Many of our choices are predictable, even if not perfectly consistent. We just call it "personality".
Authentic being (Dasein) is absolutely unpredictable. You cannot live in that state all the time, obviously, we shuttle through life on autopilot but still, we catch glimpses of it now and then when interacting with a human being (or being one:).
And I distinctly feel AI outputs lack some of that variance and unpredictability. The fractal nature of natural things also is not there with AI. It's an eery feeling, akin to the uncanny valley.
I think that any good writing is a reflection of that Dasein and reading AI outputs I am convinced there is no 'soul' in AI yet.
8 replies →
saw a meme graph lately showing the distribution of 3 different sources of a number from 1 to 10. random, human, ai. random is flat of course, human is a hilly landscape with low hills on all odd values above 1, with 7 the highest, ai is all 7.
4 replies →
A human can be biased toward extinction, like all who are living out of connection with the conditions that sustain life and/or no longer living in relationship with land as kin.
It's those humans who would make and justify the use of thinking machines that are biased in the same way without accounting for the bias.
This isn't an extinction path I've seen noted in most circles.
Indeed we are, but it takes very little to train a human being. In fact, several thousand unique ones finished pre training as I typed this comment! That allows for a considerable natural diversity compared to a chatbot.
Yes but we're all biased differently. It's a rich palette.
LLM force humans to think like machines far more than just writing code.
Writing code is a creative activity with large number of degrees of freedom and opportunities to express yourself. Button pushing and approving code changes is a deeply limiting activity that alienates you from your work.
That reminds me of a quote from Charlie Chaplin's The Great Dictator:
> Don't give yourselves to these unnatural men. Machine men with machine minds and machine hearts! You are not machines! You are not cattle! You are men! You have the love of humanity in your hearts! You don't hate! Only the unloved hate. The unloved and the unnatural! Soldiers! Don't fight for slavery! Fight for liberty!
LLM's are probabilistic but with a very limited range. They are tuned to be predictable enough to serve as a consumer product, so most ideas you get from them will revolve around the same concepts repeatedly.
People say this like it's a fundamental feature of all machine intelligence. We've bread deliberately docile golden retriever AI. That doesn't mean that we couldnt do the opposite.
now I'm trying to imagine what artificially intelligent golden retriever bread looks like
Unfortunately, mere nondeterminism still admits unsatisfactorily constrained output distributions. Training these models imbues them with certain inherent attractors, some good, some bad.
We see some of these attractors across most or all frontier models due to similarities in datasets and training methods. We are seeing a digital manifestation of Jung's collective unconscious. And we have seen just how reinforcing archetypes are in human behavior over time. Sometimes it's good, sometimes it's pretty bad.
For one, it has turned us into socially programmable machines who can be manipulated quite easily by associating desired thoughts and behavior with certain archetypes: the patriarchal/matriarchal nature of US politics is a great example of this. The same evolutionary predispositions which causes children to listen and adhere to their parents is used to cause adults to listen and adhere to their politicians.
Given the nature of the core datasets these models are trained on, we should assume they are also vulnerable to such archetype-coded social manipulation, and this is troubling in a post-singularity world: When raising domesticated rats, I learned that I had to be careful what I fed them. Rats in the wild are careful about eating new substances. They will nibble and wait a while, see how it affects them or other rats, before committing to larger bites. Domesticated rats, however, will often blindly eat anything you feed them because they have learned to trust you as a safe source of food. We are going to have this exact relationship with our digital agents.
positive effect? "very probably" -not-.
Heidegger’s conception of machines and technology is essentially nineteenth-century. Today, the concept of the "machine" is universal and qualitatively entirely different, (chaos and complexity) to the point that Heidegger would likely be left catatonic. (I do not share Heidegger’s critique of technology, as being technological is an inherent part of human nature; even in the Neolithic era, humans were already technological.)
However, the underlying critique, the essence of the "Butlerian Jihad", might be read as: at what point does a tool remain merely a tool, and at what point does it become an end in itself? Knowledge, strength, power, technology, and machines are all tools; they ought to be utilized to contribute to progress. Yet, the very concepts of evolution, progress (and the environment) are elusive. Herbert offers his own interpretation, a task that required an entire book.
> at what point does a tool remain merely a tool, and at what point does it become an end in itself?
As technological animals, our tools are IMO never "merely" tools - we adapt ourselves to them at least as much as we ask them to work for us. E.g., the process of making art is, often, the 2-way process of integrating yourself with your tools. (Maybe art is just the point on the spectrum where the use of the tool is in itself the point of the tool?)
And anyway, every occupation (professional and otherwise) that I can think of fetishizes its tools, at least a little. Or, in a different direction, we might consider how the nature of policing has changed as they've been given more and more military-style tools.
I understand; tools arise from human needs, and therefore reflect and follow human nature (good or bad, and a little fetishizing in the good side probably..). Adapting to tools is essentially learning to use them better after their creation, and refining that until tools becomes an extension of oneself (transcending it) is a "healthy" god thing. It is much like an athlete learning to surpass their own limitations (the body itself is the tool). This is entirely different from merely becoming a slightly more intelligent appendage of the tool, learning and adapting does not mean trapping oneself within a limitation; i feel that one can always change the tool, the technique, or even the goal.
Maybe people are deterministic as well? We have certain physical arrangement of chemical pathways in our systems, so maybe free will is a complete illusion.
Just because we ponder about our output and choices and retrospect does not in itself mean that it was non-deterministic.
> Another, more subtle justification for the Butlerian Jihad is also found in Frank Herbert's original novels, specifically Heidegger's thesis that the use of technology trains humans to think like machines.
Mechanistic thinking started a few centuries back:
> The historian of science E.J. Dijksterhuis memorably characterized the scientific revolution of the early modern period as a “mechanization of the world picture.” What did this amount to? And what was the character of the previous world picture this mechanization replaced?
> The medievals, following common sense and Aristotle alike, took the world to comprise a vast diversity of irreducibly different kinds—water, stone, gold, lead, grass, trees, birds, fish, clouds, stars, planets, people, and so on. Each was a substance sharply demarcated from other substances. Each had a defining nature or essence that united it to other members of its kind and set it apart from other kinds. Each had, by virtue of its nature, a distinctive teleology or final causality—a built-in directedness toward certain ends. Water by its nature was directed toward being liquid at room temperature, stone toward being solid. Trees by nature were directed toward sinking roots into the ground, and birds toward building nests. Human beings, by nature rational animals, were directed toward knowing the natures of things.
[…]
> The “mechanization” referred to by Dijksterhuis entirely subverted this picture of the world. Water, stone, trees, and birds were no longer seen as sharply demarcated substances, but instead as mere aggregates of unobservable particles that are all fundamentally of the same kind. The differences between everyday objects were no longer seen as reflecting distinct crisply defined essences, but rather as relatively superficial variations in the ways these homogeneous particles might be arranged. Many of the qualities that everyday perception presents its objects as having (color, sound, taste, odor, heat, and cold as we ordinarily experience them) were reinterpreted as mere projections of the human mind, resembling nothing in the objects themselves.
[…]
> In short, for the mechanistic conception of nature, the physical world is made up of nothing more than colorless, odorless, soundless, tasteless, meaningless particles in motion, pushing each other about to no purpose, in lawlike patterns by which more complex objects and systems of objects are generated. Science affords us a mathematically rigorous description of this mechanistic order of things, by which its behavior can be predicted and its hidden regularities uncovered and exploited for technological application.
* https://claremontreviewofbooks.com/ghosts-in-the-machine/
> Early modern philosophy and science are often said to have put a “mechanical” or “mechanistic” conception of nature at the center of Western thought. Robert Boyle referred to it as “the mechanical philosophy.” Historian of science E. J. Dijksterhuis characterized it as a “mechanization of the world picture.” Tim Crane calls it “the mechanical world picture.” But what does a mechanical or mechanistic conception of the world amount to?
> Dijksterhuis’s book The Mechanization of the World Picture surveys the history of the period during which this conception rose to hegemony, and in the Epilogue he considers several possible interpretations that the survey suggests. First, it is commonly said that a mechanistic conception of the natural world is one which sees it as a kind of machine, analogous to a clock. And such metaphors are, he says, indeed frequent in writers of the period.
* https://edwardfeser.blogspot.com/2026/06/what-is-mechanical-...
Ok this fascinating. I happen to have some Heidegger files, and went a-greppin. I'd say he's really talking about "inaminate objectness", as opposed to our human animate subjectness. Reasons:
1. In Science and Reflection, he says that both the 'completely calculated' nature of classical mechanics and the 'statistical character' of quantum mechanics are two representation of an underlying 'objectness of material nature'.
2. In Discourse on Thinking, he's far more concerned with the instrumental, utilitarian nature of machines than literal determinism, a concept he (somewhat confusingly/uniquely) connects to computation/'computational thinking', as opposed to the inherently-human capability for 'meditative thinking'. I personally would say this bit is refuted by LLMs in the end, but because they have intuitive rationality, a capability that seemed impossible to artificially design to the scientists of the time -- not because they involve dice.
3. In What is Thinking?, he gives the closest connection of this dichotomy to one of his better-known main topics, namely [handi]craft as a special subset of work: 'what maintains and sustains even this handicraft is not the mere manipulation of tools, but the relatedness to wood.' In this sense machines would be merely manipulating tools, stochastic or otherwise.
4. Overall I get the sense from skimming that lecture that his concern is a little mischaracterized, and in a kind of beautiful way that backs up my "Dune is still about AI" thesis described in a top-level comment somewhere below: his concern is about any society that could build complex machines (a 'technological' one), not the behavioral aspects of the machines themselves. So that concern applies equally to Star Trek (AGI), 1984 (no AGI), and our current post-singularity society (it's complicated/potentially ""just"" guessing).
All of this fits very well into his central thesis, which is continuing the Kantian & Hegelian tradition of examining how humans might be moral, free, and spiritual despite being meat computers of a largely-unknowable nature. In Heideggerland, this is expressed best in dramatic, nigh-unreadable passages like:
TL;DR: He'd probably think they're still just machines, and the question isn't really sound in the first place. IMHO!