People talk about a Butlerian Jihad against AI as if you could just ban LLMs and be done. I bet some govermenst would like to do that for ordinary people. They can ban visible products (chatbots, public APIs, GPAI services), and laws are already targeting those.
but you cant ban the math!
the same transformer/attention ideas work great as compressors, function approximators, and surrogate models in physics, chemistry, CFD, etc., where they show up as "PDE‑Transformer", "Neural Operator", or "Hybrid Surrogate Model", and not as "chatbot". ;-)
So even if you outlaw certain AI uses, the core tech will just move into scientific and engineering workflows under different names, where most people won’t recognize it. Would be interesting whether it is possible to write a LLM-like program just using compression and function interpolation algoritms.
> People talk about a Butlerian Jihad against AI as if you could just ban LLMs and be done.
I don't think we're talking about bans. The word Jihad, in Islam means struggle and usually it's accompanied by a religious "war", but not necessarily violent (see below).
> jihad , In Islam, the central doctrine that calls on believers to combat the enemies of their religion. According to the Qurʾān and the Ḥadīth, jihad is a duty that may be fulfilled in four ways: by the heart, the tongue, the hand, or the sword. The first way (known in Sufism as the “greater jihad”) involves struggling against evil desires. The ways of the tongue and hand call for verbal defense and right actions. The jihad of the sword involves waging war against enemies of Islam. [0]
Maybe people can't win against machines, but they sure can fight e.g. by refusing to interact with AI online, at their job, vandalizing mass surveilance cameras, etc. etc.
I believe the Butlerian Jihad caused a ban for "thinking machines" with a tenet of "Thou shalt not make a machine in the likeness of a human mind." This to me at least points at a ban for things that behave or feel like a human, not on the underlying math. An ornithopter probably has some fancy AI-esque flight stabilization, but it isn't expressed in a human way. The same goes for function approximators and surrogate models, no one would do the things we see people doing with AIs today, letting them talk them into heinous acts or fall in love with them.
That being said I can very much recommend the two Hyperion books for a good look at AI and co-dependence in sci-fi.
It'll have flight stabilization and controls and the like, but those things predate and / or do not need AI; AI algorithms are too expensive and slow for realtime flight control things. Some of those were traditionally done mechanically, even.
The Dune series doesn't stop with the first novel, I encourage all who liked the first one to continue reading up through at least God Emperor of Dune. Something the usually ironic calls for a Butlerian Jihad against modern LLMs miss is that even in the Dune universe, it just traded one form of stagnation and tyranny for another, it didn't really improve things. It delayed the threat of human extinction from a race of sentient machines, but such a future is inevitable without the Golden Path, and over the entire time groups like the Ixians were still developing technology to skirt the line.
I believe that at some point we are going to see a very serious public backlash to AI -- not because of the technology itself, but because of the way it is and will be used by large corporations to suck the life out of middle income earners, and used by governments to exert more control and suppress dissent (see China's use of AI to predict who might be likely to rebel; you can be sure the NSA is not far behind).
The last time there was serious public demonstrations against a technology was the anti-nuclear demonstrations in the 60s/70s when people were rightfully afraid that we had created something capable of destroying us all. The movements petered out, but it did force things like SALT, more aggressive IAEA to prevent proliferation, etc.
This is the classic issue of creating newer, better, more advanced technology and science within a broken social / economic system. Feeding better tools into a broken society just makes things worse. Giving dictators better ways to spy on and dominate their population (and to wage war) just worsens those things, even though that same technology and science in an already-existing utopian society could have been used to improve life on Earth for everyone.
Wouldn't surprise me if the rise of AI leads to some sort of Khmer Rouge / Cambodia-lite type of situation, where anti-technology and anti-intellectualism will grow.
Right now the message is more or less: If you want to "AI-proof" yourself, pick up a trade or manual job.
If we create a society where all the peons will work in healthcare / trades / manual labor, while the trillionaires control all their other aspects of (digital) life, that could very well usher in reactionary leaders.
I'd expect violent anti-technology rebels to be easily defeated by AI-powered drones, though.
Much like the traditional Samurai of Japan got crushed [0] by a new model peasant army of the Meiji Emperor, which adapted European guns, artillery and drill and basically exterminated them.
He was also warning against the followers, myths, institutions, and power structures that turn a talented person into an untouchable savior. Sounds familiar?
Sounds like Warhammer 40K where the God-Emperor was like "I am not the Messiah!" and everyone's like "He IS the Messiah!" but then perhaps his intent was to be worshiped as a god all along. 4d chess etc.
Much as I liked the books, enjoyed the most recent movies, big fan of Herbert; lining up the Pandora books to reread next. Etc.... people tend to read too much into his world building. Mostly he was just looking for backstory to lightly justify the story he wanted to tell. Pretty much none of the world building holds up to more than a passing scrutiny, and the Butlerian Jihad is no exception. Better to just enjoy the tales for what they are.
Agreed. He didn't have any strong opinions on AI, the idea that AI is the big bad mostly comes from his son's sequels as far as I can tell. Herbert set up the story with computers mostly written out because it's humans who are interesting, and it was humans he wanted to write a story about.
Totally agree with all the above: the jihad + OCB's role (IMO) is great author's device for setting up the scattering and evolution of the Honored Matres (chaotic evil) - where the potent mix of religion and gene manipulation become the key instruments of control by fear and seduction.
The Bene Gesserit do have prescience, what are you talking about? It's why the god emperor later tries to breed humans who's actions cannot be foreseen and can therefore escape the oppression of the future-seeing Bene Gesserit.
I'm sorry, you must be mistaken. The Bene Gesserit do not have prescience. Further more, the God Emperor plan had two part: first, to encourage the Scattering and second to make sure humans are not visible to any and all prescients.
Of course the article mixes things up in order to push a cheap narrative. Herbert's Butlerian Jihad was a rehash of Islamic ban on images of people that is supposed to prevent idolatry and preserve uniqueness of God. It's purely religious in nature and has nothing to do with "tools of oppression". Not to mention that absolutely any automation, even the simplest kind, was considered a thinking machine and was banned, not just that arbitrary range of things called AI.
This is a flattening of Herbert's angle - it is explicitly stated in the quoted section in the article that part of the reason for the ban is that the people who control thinking machines would exert outsized control on those who depend on them. This is a recurring theme in Dune, as we see with anxieties about dependency on mentats and Bene Gesserit truthsayers (the latter of which are in fact exerting hidden influence from their positions of trust).
Suk doctors are theoretically safer because of their imperial conditioning, but even that can be tampered with, as we see with Dr Yueh. This is a central problem in Dune: Whatever you depend on will gain power over you (or whoever controls the thing you depend on). Dependency on spice, on truthsayers, on mentats, or on thinking machines - even specific relationships. All of these are systemic vulnerabilities and therefore potential attack vectors in Dune.
(Edit: Suk Doctors get imperial conditioning, not mentats)
> The mentats are theoretically safer because of their imperial conditioning, but even that can be tampered with, as we see with Dr Yueh
Dr Yueh was not a mentat, but a suk doctor who was subject to conditioning. (Which was broken by the Piter de Vries, via the pain amplifiers applied to his wife, Wanna.)
Paul himself was being trained to be a mentat, and there were no hints of conditioning there, neither with Paul, Hawat, or de Vries (albeit he was described as a "twisted mentat", whatever that means).
The tools of oppression reading is backed directly by the books: "But that only permitted other men with machines to enslave them"
Is it also true that Fremen are canonically descendants of Zensunni practitioners, which is a mix of Zen and Sunni Islam
Many other restrict machines, and the dogma is encoded on a religious text called the Orange Catholic Bible. And, this might be a big jump but, some of the specific anti-machine quotes are similar to Matthew 12:31-32 IMO
Being a human may be sufficient (with important exceptions) to play chess, but playing chess well is not sufficient to be a human.
Likewise with math, it turns out, it appears. Likewise with most other things we classify as needing intelligence, I bet. It turns out that intelligence is not the same as humanity or sentience etc..
Doesn't it seem implausible that in the world of Dune they reached their level of technology without AGI? Perhaps all their tech is pre Butlerian Jihad, but even that seems questionable.
Seems like if it was adopted as a policy, there would have to be many gray areas where people pretend that they aren't really using AI - the ship's computer is just a domain specific finetuned version, after all, etc.
Additionally, the transhumanist ambitions represented by the Bene Gesserit or Navigator seem on par with all AI dangers to me. Why would the dangers of an incomprehensible machine mind be worse than the dangers of a transhuman being that was bred and optimized for generations? Who solved the alignment problem for these superhumans?
>Doesn't it seem implausible that in the world of Dune they reached their level of technology without AGI?
This implies their technology is unfathomably complex, too complex for the human mind to understand. Doesn't seem to be the case.
>Seems like if it was adopted as a policy, there would have to be many gray areas where people pretend that they aren't really using AI - the ship's computer is just a domain specific finetuned version, after all, etc.
Ships are using Navigators, who are trained in "higher mathematics" and are also prescient. They have no need for AIs, nor do they wish to court destruction if they are found out (I can only imagine the BG will have the Spacing Guild by the balls if they knew).
> Why would the dangers of an incomprehensible machine mind be worse than the dangers of a transhuman being that was bred and optimized for generations? Who solved the alignment problem for these superhumans?
The books address these questions, especially the one about a certain God-Emperor...
The Bene Gesserit tried to solve alignment through their breeding schedule, but Lady Jessica threw a wrench into it by bearing a son (Paul) out of love for Duke Leto. Paul wasn't aligned with the Bene Gesserit, and hilarity ensued.
So I agree, I don't see why thinking machines are a worse alignment problem than enhanced humans. See also the Wrath of Khan.
I try not to think too deeply about it, or I won't enjoy the story. How could a society have gravity suspensors, "shields", and lasguns, but no advanced computing capability somewhere? And why use swords/knives for fighting, when projectile weapons are so much more deadly? I still love the books, but I have friends who can't enjoy the Dune world due to its many contradictions.
Edit: I realize that "shields" supposedly made ordinary projectile weapons ineffective - but they had advanced drone technology (e.g. the assassin drone used against Paul, which was also stupid)
They had advanced computing, a whole region was full of it: IX.
The rest of the empire was too feudal and too religious to dabble into advanced technology. And the other players (the BG, the Tleilalxu, the Guild) were against it as well.
What good are drones, except as weapons of assassination? Can't use them in an conventional war, there are shields. And they're very expensive, remember, the Great Houses don't have advanced manufacturing capabilities (that's IX). And, finally, there are no great wars in the empire, most houses keep to their planetary fief; the Baron's attack on the Atreides' forces on Dune was an exception. It was also mindbogglingly expensive, 80 years of spice profit.
You're right about the suspension of disbelief necessary to enjoy the story, of course. I just felt like nitpicking because people seem to take the Butlerian Jihad seriously as a possible way to contend with AI in our real world, which I think asks too much of the story.
> Paul wasn't aligned with the Bene Gesserit, and hilarity ensued.
Hah, great perspective and I agree. Even normal humans are so difficult to align, no wonder it went wrong with a guy with prescient powers.
Humans come vaguely prealigned due to whatever is encoded in genes and also due to limitations of human bodies that put important constraints on individuals (e.g. no infinitely copyable trusted subagents). Even if you made them superhuman in some aspects a lot of that would still remain.
It seems unlikely that minds constructed by a different process would end up humanlike because they lack the evolutionary path-dependencies that shaped humans.
Current models appear somewhat human due to imitation learning/pretraining, but A) this could be deceptive as we don't know what's going on inside B) history has shown that that imitation learning becomes unnecessary once RL becomes good enough (e.g. AlphaGo -> AlphaZero), meaning we might end up with minds created from random initialization.
The Chinese government cares a lot about civil unrest. Which is why they are being much more careful with AI. They remember how the CCP came into power.
More concerned about America. Its freedom of speech means that every douchebag billionaire tech bro is allowed to run of their mouth and rile up the proles.
Note that the article doesn’t say that AI itself would do the enslavement; on the contrary:
> With the rise of AI came the rise of a technocratic class that created and controlled these machines, leading to oppressive structures over knowledge and the economy.
It also quotes Dune, where it is said more directly:
> Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.
I really despise when the term technocrat is misused. It has a very specific meaning in politics and is nowhere near "broligarch", "plutocrat from the tech industry", or any of the thousand different, legitimate ways to describe what people sometimes mean when they mistakenly say "technocrat". How can a political opinion be taken seriously when this extremely basic level of literacy on the topic is missing?
When "guess with some magic" can solve long-standing problems in mathematics that no human had been able to, it seems fair to ask whether it's approaching risky levels of intelligence.
I mean we've been using technology to solve long-standing problems in mathematics that no human had been able to do for years. Is Deep Blue sentient because it can beat any human at chess?
The people complaining about becoming stupider because of their AI usage might have something to say. Those that wish they didn't have to use AI but are coerced by their workplace might disagree. What of the day something negative happens to you based on the wrong, unreviewed decision without appeal by an AI agent? Do you truly believe this day will never come?
Man it annoys me to no end that this basic fact is so under discussed.
It’s as if calling something AI is all that’s needed to justify it being the same thing as Skynet, Dune automation, Terminators, Neuromancer, and every other cultural trope using the label of AI.
The ultimate fact is that LLMs are a pretty powerful technology that has little to nothing to do with most fictional depictions of intelligent machines.
Yesterday's plane making a turnaround because of a Bluetooth speaker with the name "bomb" (media says "a 4 letter word") makes me think the Skynet-level destruction is a risk, because humans are already too chicken-shit to take responsibility for decisions. They'd rather feel safe by just following the procedure, however stupid it is. And in the the future the procudre might be "Do whatever the AI says".
We don't have "Skynet, Dune automation, Terminators, Neuromancer, and every other cultural trope using the label of AI" today simply because LLM technology doesn't seem to directly lead to those, but the end goal of AI research is the creation of those.
It annoys me to no end that people keep looking at the finger while pointing at the moon. No, we don't have Skynet today, but you can believe Sam Altman would have loved to see Skynet appear out of ChatGPT. That is the end goal, and it's worth discussing now, not when it's too late.
People talk about a Butlerian Jihad against AI as if you could just ban LLMs and be done. I bet some govermenst would like to do that for ordinary people. They can ban visible products (chatbots, public APIs, GPAI services), and laws are already targeting those.
but you cant ban the math!
the same transformer/attention ideas work great as compressors, function approximators, and surrogate models in physics, chemistry, CFD, etc., where they show up as "PDE‑Transformer", "Neural Operator", or "Hybrid Surrogate Model", and not as "chatbot". ;-)
So even if you outlaw certain AI uses, the core tech will just move into scientific and engineering workflows under different names, where most people won’t recognize it. Would be interesting whether it is possible to write a LLM-like program just using compression and function interpolation algoritms.
just my 2 ct
> People talk about a Butlerian Jihad against AI as if you could just ban LLMs and be done.
I don't think we're talking about bans. The word Jihad, in Islam means struggle and usually it's accompanied by a religious "war", but not necessarily violent (see below).
> jihad , In Islam, the central doctrine that calls on believers to combat the enemies of their religion. According to the Qurʾān and the Ḥadīth, jihad is a duty that may be fulfilled in four ways: by the heart, the tongue, the hand, or the sword. The first way (known in Sufism as the “greater jihad”) involves struggling against evil desires. The ways of the tongue and hand call for verbal defense and right actions. The jihad of the sword involves waging war against enemies of Islam. [0]
Maybe people can't win against machines, but they sure can fight e.g. by refusing to interact with AI online, at their job, vandalizing mass surveilance cameras, etc. etc.
[0] - https://www.britannica.com/summary/jihad
I believe the Butlerian Jihad caused a ban for "thinking machines" with a tenet of "Thou shalt not make a machine in the likeness of a human mind." This to me at least points at a ban for things that behave or feel like a human, not on the underlying math. An ornithopter probably has some fancy AI-esque flight stabilization, but it isn't expressed in a human way. The same goes for function approximators and surrogate models, no one would do the things we see people doing with AIs today, letting them talk them into heinous acts or fall in love with them.
That being said I can very much recommend the two Hyperion books for a good look at AI and co-dependence in sci-fi.
It'll have flight stabilization and controls and the like, but those things predate and / or do not need AI; AI algorithms are too expensive and slow for realtime flight control things. Some of those were traditionally done mechanically, even.
> Would be interesting whether it is possible to write a LLM-like program just using compression and function interpolation algoritms.
gzip can be used as a (not very good) LLM-like text and image generator: https://arxiv.org/abs/2309.10668
You can ban the math by limiting the energy and the material resources.
The Dune series doesn't stop with the first novel, I encourage all who liked the first one to continue reading up through at least God Emperor of Dune. Something the usually ironic calls for a Butlerian Jihad against modern LLMs miss is that even in the Dune universe, it just traded one form of stagnation and tyranny for another, it didn't really improve things. It delayed the threat of human extinction from a race of sentient machines, but such a future is inevitable without the Golden Path, and over the entire time groups like the Ixians were still developing technology to skirt the line.
I believe that at some point we are going to see a very serious public backlash to AI -- not because of the technology itself, but because of the way it is and will be used by large corporations to suck the life out of middle income earners, and used by governments to exert more control and suppress dissent (see China's use of AI to predict who might be likely to rebel; you can be sure the NSA is not far behind).
The last time there was serious public demonstrations against a technology was the anti-nuclear demonstrations in the 60s/70s when people were rightfully afraid that we had created something capable of destroying us all. The movements petered out, but it did force things like SALT, more aggressive IAEA to prevent proliferation, etc.
This is the classic issue of creating newer, better, more advanced technology and science within a broken social / economic system. Feeding better tools into a broken society just makes things worse. Giving dictators better ways to spy on and dominate their population (and to wage war) just worsens those things, even though that same technology and science in an already-existing utopian society could have been used to improve life on Earth for everyone.
despite being so smart, humans are so f*ing stupid
Wouldn't surprise me if the rise of AI leads to some sort of Khmer Rouge / Cambodia-lite type of situation, where anti-technology and anti-intellectualism will grow.
Right now the message is more or less: If you want to "AI-proof" yourself, pick up a trade or manual job.
If we create a society where all the peons will work in healthcare / trades / manual labor, while the trillionaires control all their other aspects of (digital) life, that could very well usher in reactionary leaders.
I'd expect violent anti-technology rebels to be easily defeated by AI-powered drones, though.
Much like the traditional Samurai of Japan got crushed [0] by a new model peasant army of the Meiji Emperor, which adapted European guns, artillery and drill and basically exterminated them.
[0] https://en.wikipedia.org/wiki/Satsuma_Rebellion
[dead]
He was also warning against the followers, myths, institutions, and power structures that turn a talented person into an untouchable savior. Sounds familiar?
Sounds like Warhammer 40K where the God-Emperor was like "I am not the Messiah!" and everyone's like "He IS the Messiah!" but then perhaps his intent was to be worshiped as a god all along. 4d chess etc.
Much as I liked the books, enjoyed the most recent movies, big fan of Herbert; lining up the Pandora books to reread next. Etc.... people tend to read too much into his world building. Mostly he was just looking for backstory to lightly justify the story he wanted to tell. Pretty much none of the world building holds up to more than a passing scrutiny, and the Butlerian Jihad is no exception. Better to just enjoy the tales for what they are.
Agreed. He didn't have any strong opinions on AI, the idea that AI is the big bad mostly comes from his son's sequels as far as I can tell. Herbert set up the story with computers mostly written out because it's humans who are interesting, and it was humans he wanted to write a story about.
Totally agree with all the above: the jihad + OCB's role (IMO) is great author's device for setting up the scattering and evolution of the Honored Matres (chaotic evil) - where the potent mix of religion and gene manipulation become the key instruments of control by fear and seduction.
>the Bene Gesserit sisterhood with their powers of prescience
no, the Bene Gesserit doesn't have prescience, the opposite maybe (postcognition)
>Spice melange, which can only be found on the planet Arrakis (aka Dune), is the drug that enables this enhanced mental cognition
no, for anyone not called Atreides spice just prolongs life; some mentats take sapho juice, but they don't need drugs. .
As I recall, perhaps erroneously, Guild Navigators are nominal humans who owe their ability to move ships to being spiced to their gills.
Yes, I forgot about the Guild Navigators. Technically they don't move the ships, they see the safe path to the destination.
1 reply →
I think the navigators were also gmo'ed a bit
The Bene Gesserit do have prescience, what are you talking about? It's why the god emperor later tries to breed humans who's actions cannot be foreseen and can therefore escape the oppression of the future-seeing Bene Gesserit.
I'm sorry, you must be mistaken. The Bene Gesserit do not have prescience. Further more, the God Emperor plan had two part: first, to encourage the Scattering and second to make sure humans are not visible to any and all prescients.
Of course the article mixes things up in order to push a cheap narrative. Herbert's Butlerian Jihad was a rehash of Islamic ban on images of people that is supposed to prevent idolatry and preserve uniqueness of God. It's purely religious in nature and has nothing to do with "tools of oppression". Not to mention that absolutely any automation, even the simplest kind, was considered a thinking machine and was banned, not just that arbitrary range of things called AI.
This is a flattening of Herbert's angle - it is explicitly stated in the quoted section in the article that part of the reason for the ban is that the people who control thinking machines would exert outsized control on those who depend on them. This is a recurring theme in Dune, as we see with anxieties about dependency on mentats and Bene Gesserit truthsayers (the latter of which are in fact exerting hidden influence from their positions of trust).
Suk doctors are theoretically safer because of their imperial conditioning, but even that can be tampered with, as we see with Dr Yueh. This is a central problem in Dune: Whatever you depend on will gain power over you (or whoever controls the thing you depend on). Dependency on spice, on truthsayers, on mentats, or on thinking machines - even specific relationships. All of these are systemic vulnerabilities and therefore potential attack vectors in Dune.
(Edit: Suk Doctors get imperial conditioning, not mentats)
> The mentats are theoretically safer because of their imperial conditioning, but even that can be tampered with, as we see with Dr Yueh
Dr Yueh was not a mentat, but a suk doctor who was subject to conditioning. (Which was broken by the Piter de Vries, via the pain amplifiers applied to his wife, Wanna.)
Paul himself was being trained to be a mentat, and there were no hints of conditioning there, neither with Paul, Hawat, or de Vries (albeit he was described as a "twisted mentat", whatever that means).
2 replies →
The tools of oppression reading is backed directly by the books: "But that only permitted other men with machines to enslave them"
Is it also true that Fremen are canonically descendants of Zensunni practitioners, which is a mix of Zen and Sunni Islam
Many other restrict machines, and the dogma is encoded on a religious text called the Orange Catholic Bible. And, this might be a big jump but, some of the specific anti-machine quotes are similar to Matthew 12:31-32 IMO
Not a very deep justification. Probably the easiest thing to grasp for when thinking "how can I make this religious theme fit the narrative?".
3 replies →
Seems a bit premature given the current state of word generators.
I guess top mathematicians are also mere word generators?
https://openai.com/index/model-disproves-discrete-geometry-c...
Being a human may be sufficient (with important exceptions) to play chess, but playing chess well is not sufficient to be a human.
Likewise with math, it turns out, it appears. Likewise with most other things we classify as needing intelligence, I bet. It turns out that intelligence is not the same as humanity or sentience etc..
1 reply →
They’re useful tools, they’re not general AI, nor are they anywhere near the scenario in Dune.
If you're going to do a jihad it'll be easier against ChatGPT than waiting till they have Terminator 2 style things.
If you want to base your core beliefs on sci-fi books at least read a little more than Dune.
Go read some books from The Culture, better series than the Dune.
Excellent counter example, since The Culture books portray a utopia made possible by truly benevolent AI overlords.
I think it is unlikely that people will act to completely abolish technology once they are used to it in their everyday lives.
This is called technological momentum.
https://en.wikipedia.org/wiki/Technological_momentum
The Amish (can't) enter the chat
Ok, you made me laugh :)
I think that most Amish were never used to technology though?
2 replies →
The article had a great opportunity to at least reference the newest encyclical from the Pope.
With a bit of click bait, this might be the first step of "Butlerian Jihad" -> "AI Crusade" and a founding text for the Orange CATHOLIC Bible.
I saw a take that you can now cite religious reasons to refuse working on and with AI - if more people try this, I wonder how it'd play out.
Interesting times.
Doesn't it seem implausible that in the world of Dune they reached their level of technology without AGI? Perhaps all their tech is pre Butlerian Jihad, but even that seems questionable.
Seems like if it was adopted as a policy, there would have to be many gray areas where people pretend that they aren't really using AI - the ship's computer is just a domain specific finetuned version, after all, etc.
Additionally, the transhumanist ambitions represented by the Bene Gesserit or Navigator seem on par with all AI dangers to me. Why would the dangers of an incomprehensible machine mind be worse than the dangers of a transhuman being that was bred and optimized for generations? Who solved the alignment problem for these superhumans?
>Doesn't it seem implausible that in the world of Dune they reached their level of technology without AGI?
This implies their technology is unfathomably complex, too complex for the human mind to understand. Doesn't seem to be the case.
>Seems like if it was adopted as a policy, there would have to be many gray areas where people pretend that they aren't really using AI - the ship's computer is just a domain specific finetuned version, after all, etc.
Ships are using Navigators, who are trained in "higher mathematics" and are also prescient. They have no need for AIs, nor do they wish to court destruction if they are found out (I can only imagine the BG will have the Spacing Guild by the balls if they knew).
> Why would the dangers of an incomprehensible machine mind be worse than the dangers of a transhuman being that was bred and optimized for generations? Who solved the alignment problem for these superhumans?
The books address these questions, especially the one about a certain God-Emperor...
The Bene Gesserit tried to solve alignment through their breeding schedule, but Lady Jessica threw a wrench into it by bearing a son (Paul) out of love for Duke Leto. Paul wasn't aligned with the Bene Gesserit, and hilarity ensued.
So I agree, I don't see why thinking machines are a worse alignment problem than enhanced humans. See also the Wrath of Khan.
I try not to think too deeply about it, or I won't enjoy the story. How could a society have gravity suspensors, "shields", and lasguns, but no advanced computing capability somewhere? And why use swords/knives for fighting, when projectile weapons are so much more deadly? I still love the books, but I have friends who can't enjoy the Dune world due to its many contradictions.
Edit: I realize that "shields" supposedly made ordinary projectile weapons ineffective - but they had advanced drone technology (e.g. the assassin drone used against Paul, which was also stupid)
They had advanced computing, a whole region was full of it: IX.
The rest of the empire was too feudal and too religious to dabble into advanced technology. And the other players (the BG, the Tleilalxu, the Guild) were against it as well.
What good are drones, except as weapons of assassination? Can't use them in an conventional war, there are shields. And they're very expensive, remember, the Great Houses don't have advanced manufacturing capabilities (that's IX). And, finally, there are no great wars in the empire, most houses keep to their planetary fief; the Baron's attack on the Atreides' forces on Dune was an exception. It was also mindbogglingly expensive, 80 years of spice profit.
3 replies →
You're right about the suspension of disbelief necessary to enjoy the story, of course. I just felt like nitpicking because people seem to take the Butlerian Jihad seriously as a possible way to contend with AI in our real world, which I think asks too much of the story.
> Paul wasn't aligned with the Bene Gesserit, and hilarity ensued.
Hah, great perspective and I agree. Even normal humans are so difficult to align, no wonder it went wrong with a guy with prescient powers.
Humans come vaguely prealigned due to whatever is encoded in genes and also due to limitations of human bodies that put important constraints on individuals (e.g. no infinitely copyable trusted subagents). Even if you made them superhuman in some aspects a lot of that would still remain. It seems unlikely that minds constructed by a different process would end up humanlike because they lack the evolutionary path-dependencies that shaped humans. Current models appear somewhat human due to imitation learning/pretraining, but A) this could be deceptive as we don't know what's going on inside B) history has shown that that imitation learning becomes unnecessary once RL becomes good enough (e.g. AlphaGo -> AlphaZero), meaning we might end up with minds created from random initialization.
Why are you assuming that some technology can only be discovered by AGI?
"Who solved the alignment problem for these superhumans?"
The gun, pointed to their head.
The Chinese government cares a lot about civil unrest. Which is why they are being much more careful with AI. They remember how the CCP came into power.
More concerned about America. Its freedom of speech means that every douchebag billionaire tech bro is allowed to run of their mouth and rile up the proles.
LLMs cannot think, they just guess with some magic with the help of things like softmax (https://en.wikipedia.org/wiki/Softmax_function), that makes it seem intelligent.
We aren't enslaved or at any risk just yet with this form of "thinking machine", but we do need to culutrally and emotionally come to terms with them.
Let's not have any other "jihad" just yet, there's plenty of other jihads going on right now.
Note that the article doesn’t say that AI itself would do the enslavement; on the contrary:
> With the rise of AI came the rise of a technocratic class that created and controlled these machines, leading to oppressive structures over knowledge and the economy.
It also quotes Dune, where it is said more directly:
> Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.
I really despise when the term technocrat is misused. It has a very specific meaning in politics and is nowhere near "broligarch", "plutocrat from the tech industry", or any of the thousand different, legitimate ways to describe what people sometimes mean when they mistakenly say "technocrat". How can a political opinion be taken seriously when this extremely basic level of literacy on the topic is missing?
5 replies →
When "guess with some magic" can solve long-standing problems in mathematics that no human had been able to, it seems fair to ask whether it's approaching risky levels of intelligence.
I mean we've been using technology to solve long-standing problems in mathematics that no human had been able to do for years. Is Deep Blue sentient because it can beat any human at chess?
2 replies →
[dead]
> We aren't enslaved
The people complaining about becoming stupider because of their AI usage might have something to say. Those that wish they didn't have to use AI but are coerced by their workplace might disagree. What of the day something negative happens to you based on the wrong, unreviewed decision without appeal by an AI agent? Do you truly believe this day will never come?
You don't need literal chains to be a slave.
Man it annoys me to no end that this basic fact is so under discussed.
It’s as if calling something AI is all that’s needed to justify it being the same thing as Skynet, Dune automation, Terminators, Neuromancer, and every other cultural trope using the label of AI.
The ultimate fact is that LLMs are a pretty powerful technology that has little to nothing to do with most fictional depictions of intelligent machines.
Sloppy thinking all the way down.
Yesterday's plane making a turnaround because of a Bluetooth speaker with the name "bomb" (media says "a 4 letter word") makes me think the Skynet-level destruction is a risk, because humans are already too chicken-shit to take responsibility for decisions. They'd rather feel safe by just following the procedure, however stupid it is. And in the the future the procudre might be "Do whatever the AI says".
My thought about the turnaround: https://news.ycombinator.com/item?id=48343390
1 reply →
People worry about what will come in the future even if today's tech has limits.
We don't have "Skynet, Dune automation, Terminators, Neuromancer, and every other cultural trope using the label of AI" today simply because LLM technology doesn't seem to directly lead to those, but the end goal of AI research is the creation of those.
It annoys me to no end that people keep looking at the finger while pointing at the moon. No, we don't have Skynet today, but you can believe Sam Altman would have loved to see Skynet appear out of ChatGPT. That is the end goal, and it's worth discussing now, not when it's too late.
2 replies →