Comment by tarpitt
3 days ago
My favorite quote from dune:
Greatness is a transitory experience. It is never consistent. It depends in part upon the myth-making imagination of humankind. The person who experiences greatness must have a feeling for the myth he is in. He must reflect what is projected upon him. And he must have a strong sense of the sardonic. This is what uncouples him from belief in his own pretensions. The sardonic is all that permits him to move within himself. Without this quality, even occasional greatness will destroy a man.
I couldn't help but notice something here. It is relevant with my writing experience/experiments with minimal assitance from LLMs (that lasted for 6–7 weeks).
> Greatness is a transitory experience. It is never consistent.
If you ever ask an LLM about these two sentences, it'd tell you that the second sentence is a mere repition of the first and quite possibly add that it "doesn't land". But that gives me hope. That there's something in those rules and algorithms manfiesting at the inhumane cost of unfathomable memory, disk space, and compute power that it will never (or for a very long time) match the human experience.
As little as I sought the help from LLMs (completely restricted to "in place" grammar and punctuation fixes and just flag it if it was not possible without sentence/phrase change) I noticed my writing was changing and not in a good way. It was not the insidiousness of these LLMs that was changing me but the nuanced incompetences.
I've read a very cool article on this exact subject recently. https://permeations.substack.com/p/i-would-have-been-killed-...
Coincidentally, also references Butlerian Jihad.
Thank you! It's cool indeed.
Language is an inseparable part of modern humans. It’s likely language evolved as an internal tool for thought and executive function, and externalization of language came afterwards as a means to transmit our thoughts to the minds of other humans.
As such, writing (another form of externalizing our internal language) is a reflection of our essential human qualities.
LLMs can certainly mimic that quite effectively, but they can’t exceed it. I think that’s why LLM writing still never quite hits the mark. It’s the right shape, but it’s missing intent. The words were not chosen to represent real thoughts in someone’s mind.
Even if AI intelligence exceeds humans, it still won’t be human.
> It’s likely language evolved as an internal tool for thought and executive function, and externalization of language came afterwards as a means to transmit our thoughts to the minds of other humans
Isn't this in contradiction to the observations of severely neglected or feral children who having never learned language as a child cannot learn it later in life?
How would any form of inner language evolve in the first place for executive function?
Communication first makes a lot more sense evolutionarily since the ability to communicate more information creates a clear improvement in fitness e.g. being able to signal danger improves survival. It is also more amenable to small improvements in capacity e.g. going from "danger" to "danger wolf" to "danger wolf behind" etc. and we observe these language primitives in other species.
Internal language seems much more likely to be a later development especially given that there are people today who claim they have none / need none and yet still possess the ability to communicate with others and manage their lives normally.
> LLMs can certainly mimic that quite effectively, but they can’t exceed it. I think that’s why LLM writing still never quite hits the mark. It’s the right shape, but it’s missing intent. The words were not chosen to represent real thoughts in someone’s mind.
Well yeah. That's because they aren't actually intelligent. They can be useful in the right circumstances despite that, but it's very important to remember that LLMs categorically cannot think, know things, or reason. They are a statistical model, that's it.
1 reply →
How are they repetition? Being transitory and being consistent are unrelated characteristics.
I mean, we have entire forms of art dedicated to saying the same thing in different words. Half of poetry, for example. Human language is redundant for a reason.
> But that gives me hope.
> It was not the insidiousness of these LLMs that was changing me but the nuanced incompetences.
AI doesn't yet improve on it's own, so it's slow at picking up all those small incompetences. We still have advantage in rate of learning, and some people are essentially specialised in learning themselves (I think that this applies to 90% of HN'ers and defines THE bubble we're in). But when I write that, I still have to remember in the back of my head, that AI knows waaaay more than me, so small incompetences I can still see are just a side effect. And I sometimes make the error and correct AI, which will probably be used for it's training in next rounds...
Yes and this is what worries me.. the not yet part.
> AI doesn't yet improve on it's own
I mean when that day arrives and that aspect of AI is not controlled from now it will be quite unfortuante for most of the humanity (and let's just assume here we are not wiped out by AI; or not even mention that).
Think of nuclear technology. Is it smarter than us or something on those lines? the answer is it doens't matter. What matters is whoever weilds this power and then limited/restricted others (this was a good thing and a bad thing because some had – at the same time) from having that power essentially had a controlling upper hand on a whole lot of humanity.
AI is something like that with a big difference that it is essentially on the way to "replace" our usefulness, our professional relevance if I can say that. We can go about long spirals of "but soemthing else" will turn up, just like we can keep going about what if nuclear tech didn't even happen, wasn't invented, or what if it was done in such a controlled manner that only it's energy and other need was possible to harness (no pun intended!). It happened and this is how it happened. That's exacty what is happening with AI.
In the hands of few. Extremely powerful. Even if there are open models, running the bigger and better ones are simply out of reach for most companies (let alone people). And yes, it is already being weaponised. It's going to be such a weapon of control that nuclear tech won't even compare. Usign nukes to subdue a nation earlier meant also destrying a a lot of shared environment in the process. But with AI and almost exclsuive frontier level weaponised access to it with deployment capabiltiy the damage can be done is such controlled manner and scale that even things like protests will become too hard to do. In fact AI wiping out the humanity is least of my worries.
> If you ever ask an LLM about these two sentences, it'd tell you that the second sentence is a mere repition of the first and quite possibly add that it "doesn't land".
I did that, and it's not what happened. I asked in several ways. Idk what you're on about.
My favourite quote in reference to rebellion against AI, and it is even quoted in TFA too:
That's from Erewhon, by Samuel Butler. TFA purposes that the term Butlerian Jihad comes specifically from this passage.
Butler also wrote a famous essay "Darwin Among The Machines" (1863) before Erewhon, which treated this concept not as fiction.
"Erewhon" oddly close to "nowhere" spelled backwards.
Hardly odd; it's intentional. There are quite a few backwards and anagram words and names in the book: https://en.wikipedia.org/wiki/Erewhon
My favorite quote from Dune, relevant to "AI":
> Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.
The techno-fascists of today. We are already there.
This is such an important point. I don't quite understand why smaller businesses are wanting to replace human staff with AI systems when that will just lead to them being held hostage by the companies that supply the AI systems.
What if they had turned their thinking over to open weight models that they could fine tune and ran on their own hardware?
Considering that the aptly named "Fremen" of the Dune Universe would gladly murder a stranger to get their body's water, and literally went into a galactic genocide that left hundreds of billions dead ...
... maybe we should take with a grain of salt every moral aphorism uttered by someone living in that universe.
This isn't from the first book, is it?
It is. From one of the chapter epigraphs.
I too, believe personality cults (NK, Nazi Germany, Shiite Iran, etc.being just the recent ones) are one of most evil social (power) structures.