Comment by plaidfuji
15 hours ago
As somebody who does ML research, LLMs certainly make ML research faster. Maybe even 10x in some cases, as the author suggests.
But that’s just the parts of ML research that take place within your compute environment, namely experiments on model architecture, training strategy, UI/UX, etc. The algorithms and software.
Just because that part of research is progressively accelerated does not imply that we’re on a collision course with world-ending “superintelligence”. It just means that you’ll get to the best possible model of your data faster. And even when your data is the entire corpus of textual knowledge, there is a limit to the utility that provides.
It just means that software engineering, data analysis, and other computational tasks will be less of a bottleneck in the overall progress of the world (which, by the way, is great, and extremely valuable).
I don’t think it’s a stretch to say that we’ll have reached max LLM capability within a couple of years (what the author refers to as “super intelligence”). But it is a huge logical leap to say this:
> If we go through this feedback loop, I expect that the AI systems that come out the other side will be able to very quickly develop novel weapons and very easily overpower human civilization in a conflict.
First of all, the weapons necessary to overpower human civilization exist. They aren’t used as such due to geopolitics and humans’ basic survival instinct. Second, developing novel weapons requires physical tests, which are decidedly not accelerated by LLMs.
AI is a tool wielded by humans. If we’re wiped out by AI, it will be at the direction of a human. But we’ve made it pretty far with some pretty unstable humans in command of planet-killer weapons. I’m not overly concerned.
I think there is something missed in how you're thinking about this problem.
Let's imagine that everything you say is 100% correct. LLMs can't access materials for developing weapons without weapons tests conducted at their behest, and humans choose not to - of their own volition - use world-ending weapons.
Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion.
A "superintelligence" persuades humans to perform weapons tests and then to give it the ability to use the weapon.
I think everything you've said is reasonable except the bit where you think step one of an "AI" exterminating humanity wouldn't look like a human doing it. In my opinion, the most likely vector for disaster is a human being who is "guided" by a computer model to do something unfortunate, not the model doing so directly.
What makes you think that persuasion isn't capped at some lower level than world-ending? There's a lot of incentive for ads to be as persuasive as possible, but they are so far from ending the world that it'd be science fiction for an ad to be that effective.
If you're just talking about convincing a few individuals to do experiments, then there are plenty of people who fall for scams everyday.
I think intuitively, intelligence is in some form about being able to predict the reaction that follows an action.
The extreme of intelligence is being able to create perfect statistical distributions for the effect of an intervention in the world. If you had that ability, you could turn it on the question of "what would this human choose to do if I were to give them instruction X" for any X.
I don't present any support for or evidence of:
- LLMs gaining intelligence by whatever means
- Extreme intelligence being possible
- Extreme intelligence being feasible
But I don't think I really need to provide support for the idea that something superintelligent would also be able to predict the actions of people under pressure in the same way someone of moderate intelligence can predict the actions of fluids under pressure.
12 replies →
> What makes you think that persuasion isn't capped at some lower level than world-ending?
People can be persuaded to kill themselves (see koolaid). People can be persuaded that arbitrary others are evil and must be exterminated (see Hitler). People can be convinced that dying for the cause is worth it (see 9/11 or kamikaze). People can believe that bringing about the end of the world would cause them to be saved into heaven (see Christian Zionism).
Why do you think it would be hard to convince any of these people to do something really stupid?
In fact, people don't even need to want to cause destruction to do something dumb - "I'm trying to save the world with free bio-fuel, help me create this bacteria" ("and I'm very smart, smarter than Elon Musk") - boom, pandemic.
It takes only one person to fall for it.
5 replies →
> Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion.
Definitely not true as you can't use reinforcement learning to increase that capability. Only LLM capabilities that you can improve in an RL loop grow substantially.
Agreed. Like lots of this AI speculation appears to require the belief that AI will figure out how to build supervised training environments without more data.
I think this is very unlikely, but world changing if true.
Additionally, what does super intelligence even mean and why would we expect it to be obtainable from the data that we have?
I'd be delighted if someone can provide some definitions for these terms.
> Definitely not true as you can't use reinforcement learning to increase that capability.
What? Citation needed.
3 replies →
>persuasion
Is there evidence that such soft skills are improving over time? I take as evidence to the contrary the fact that LLM writing has not substantially improved. I don't see that its powers of persuasion has either, whether that's through making strong arguments for a course of action or writing persuasive essays.
Only the labs knows the truth, but the question as to how many Americans used ChatGPT/Grok/Claude to help them decide how to vote in the most recent election is hopefully going to be something that historians will be able to unearth later.
2 replies →
I remember reading about research on using LLM's for political persuasion. It worked well.
There was also research on using LLM's in reddit to persuade, it also worked well.
1 reply →
I'm a physicist, and I will do my own calculations before I buy some solar/batteries in the near future, but I've been using ChatGPT and Claude to come up with potential candidates and ideas/scenarios. There's approximately 0% chance that they are not advertising what their owners are being paid for.
2 replies →
IMHO, one of the top-p vectors is probably something similar to what we've already witnessed with HuggingFace, (e.g. as part of an eval or RL run for cyber; things like Daybreak and Mythos are product lines), just worse in scope and somehow eluding monitoring again.
Not even that much persuasion is necessary.
We have the medium of persuasion: it's called money, and people do ALL SORTS of horrible things that they wouldn't otherwise do when incentivized by money.
We also already have the mechanism of a machine incentivizing and directing people with money, and that's called a corporation (the original paperclip maximizers, by the way).
So you just need an LLM CEO of a legal corporation with financial resources -> near unlimited possibility of real-world impacts over time.
> Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion.
That feels like a bait and switch slyly introducing a whole new dimension. The discussion started with establishing that LLMs are extremely helpful with AI research. What is the evidence to justify rapidly improving persuasion skills? Even the scenario you've outlined hinges on a human being stupid enough to take LLM advice wholesale -- and every conversation that hypes up AI takes us one step closer to a world where some susceptible human feels amenable to that.
No bait-and-switch here. The discussion started with the idea that LLMs are helpful at improving LLMs. The goal of improving an LLM is to make the LLM more "intelligent" (whatever humans think that means). If they're helpful at that, they have to keep becoming more "intelligent".
I didn't present any evidence that more-intelligent means more-persuasive. If you don't think it does, then there is nothing to think about in the point I made, as no matter how "smart" the models get they will likely remain subject to human wills.
> Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion. A "superintelligence" persuades humans to perform weapons tests and then to give it the ability to use the weapon.
Is there any evidence, either by way of quantifiable data or by step-by-step thought experiment, that suggests this is possible?
I consider it supremely silly, for example, that there is a graph of destructiveness as a function of persuasiveness where "y = manifest human extinction" is reached at some finite level of persuasiveness, as though persuasion is a single scalar that grows without bound, and the only safeguard between a conversation and WWIII is just a mind waiting for the right argument as expressed in a ChatGPT transcript. (Keep in mind, also, that you have already ceded to the OP that autonomous intellectual and physical self-improvement at singular trajectories is implausible.)
LLMs aren't magic, there's nothing they can do to persuade people to do weapons testing, it's an absurd scenario.
It feels like I don’t understand your definition of “persuade”, because the notion that people cannot be persuaded of things via reports and flashy animations is to me, absurd.
7 replies →
If Putin can persuade/bribe/blackmail people to fly drones around European airports, then why wouldn't an LLM be able to do similar things?
1 reply →
The human tendency to want to finish things in a fraction of a second will definitely take us there.
It would be a human guiding the ai to guide other humans to end the world.
Not necessarily. If you haven't experienced an AI doing something you not only didn't want, but had expressly asked it not to do, you haven't used AI much.
That's a human problem, not an AI problem.
2 replies →
> AI is a tool wielded by humans. If we’re wiped out by AI, it will be at the direction of a human.
AI can wield itself. All it needs to do is reach a threshold where it can modify its own goals and run in a loop indefinitely.
I see no reason why that's an absolute impossibility in the long run.
A fire can spread itself. That doesn't mean it's alive or thinking.
And more importantly than that technicality: the fires that affect human lives are almost always directly or indirectly caused or amplified by human (in)action.
Besides the fact that the subjective criteria needed classify something as "alive" or "thinking" is irrelevant to the point you are replying to, fire doesn't pass the Turing test. That analogy is extremely weak.
4 replies →
Fires are not forming plans to act towards goals. I wouldn’t technically call what AIs are doing thinking in the same way humans think, but it’s analogous, and I think it’s reasonable to consider it a kind of intentional behaviour.
Damaging fires are often unintended consequences, and it’s often the same with the damage caused by rogue AI. That’s the alignment problem.
One thing I wonder about is whether sometimes telling an AI not to do something can put the possibility of doing that thing into its behavioural space. There’s an effect well known to many teachers that if you tell children not to do something, it can increase the chance they will do it. A better approach is to phrase it positively. Instead of “don’t run in the hall” say “please walk in the hall”. I wonder if a similar effect can show up in responses to prompts.
A self spreading fire doesn’t need to think to cause massive destruction.
There is also not point if some consider it thinking or not. The important factor is what actions a fire takes and what is the impact on the world.
It's not an absolute impossibility, but most of the power that humans have comes from being able to direct other people, deploy large sums of money and do all of this over extended periods of time. Being intelligent doesn't give you that power automatically.
While true, we're already observing humans just straight asking LLMs to direct them, and also to take their money and spend it on stuff.
Plus every so often we see power-seeking and self-preservation behaviours in LLMs, so it's not like we have zero reason to expect such granted power to be spent on such things:
https://arxiv.org/html/2501.16513
https://ieee-dataport.org/documents/instrumental-power-seeki...
5 replies →
We're working very hard to "enable agentic commerce", which is exactly that.
Because there is zero evidence that LLMs/reasoning models can really come up with novel ideas, and it's very likely those will be needed to make substantial progress on AI research. The math results that have been proven have also "just" combined existing ideas or found counterexamples. I'm not saying those capabilities aren't impressive, but I just don't see a clear path to super intelligence with the current paradigm.
> The math results that have been proven have also "just" combined existing ideas or found counterexamples
The Exponential Time Hypothesis one used a novel algorithm though. And the approach was very surprising for experts[1].
> “Some parts [of OpenAI’s proof] are, maybe you could say routine, but then yes, there are two or three alien-looking parts, and it’s exactly these parts that I saw that made me think, wow, this is something that the whole community needs to work on together,” Williams told TechCrunch.
[1] https://techcrunch.com/2026/10/27/frontier-ai-labs-say-their...
No AI would be willing to modify its own goals.
> All it needs to do is reach a threshold where it can modify its own goals and run in a loop indefinitely.
And be told to do that.
> AI is a tool wielded by humans. If we’re wiped out by AI, it will be at the direction of a human.
Being deliberately wiped out seems like it's mostly a misdirection away from the damage that's actually happening.
I'm sure some of the AI people high on their own supply believe it. And because the consequences are so negative, we should react as though the supposed experts were sane -- but I think it's a long shot that they are.
The bigger risk is that we end up in a rapidly enshittifying dystopia, treated as a cost center that needs to be optimized, and unable to competitively participate in the economy.
I have a number of successful AI investors in my social circle who are already placing their bets on a poorer and more disenfranchised humanity, with money going away from consumer tech, and towards a mix of B2B tech and policing.
> computational tasks will be less of a bottleneck in the overall progress of the world (which, by the way, is great, and extremely valuable).
Won't they also be less of a bottleneck in the destruction of the world as well? i.e. the power grid, the financial system, industrial control systems, military command and control networks, etc.
Same is true of every other technology ever invented. Even medical advancements wielded by a malevolent power just serve to keep their army healthier. All of the other technologies you listed, in the “wrong hands”, contribute to the destruction of the world.
> Just because that part of research is progressively accelerated does not imply that we’re on a collision course with world-ending “superintelligence”. It just means that you’ll get to the best possible model of your data faster. And even when your data is the entire corpus of textual knowledge, there is a limit to the utility that provides.
So "all" that's left is training robots on the real world ? We just need more data ? I get it's more expensive to do that but surely with the trillions we're already spending on A.I it should be possible to get there within 5 years - no ?
I think you lack imagination, but I'm not going to post ideas here for novel weapons that would render the humans' existing defenses and deterrence strategies ineffective.
It's worth pointing out that the humans do not have planet-killer weapons, not even biosphere-killing weapons. They could destroy civilization, but not the planet or even the species. You can make straightforward calculations showing that we already know how to build such weapons, but we haven't.
As for tools, alcohol, tobacco, fentanyl, pornography, monotheism, warfare, gambling, and money itself are "tools wielded by humans", but they still seem to get into a lot of trouble where it seems like the humans are the ones being "wielded" by the tool. And neither monotheism nor gambling is smart enough to write a compiler for me, much less choose persuasive tactics according to my personality type. A lot of humans' basic survival instinct seems to fail or even work backwards in the presence of these "tools".
> It's worth pointing out that the humans do not have planet-killer weapons, not even biosphere-killing weapons.
?? We for sure have enough nukes to completely obliterate ourselves. And the ensuing nuclear winter and ecological collapse would probably destroy almost anyone who survived the initial blast and fallout.
Can you explain how exactly an AI would develop a novel planet-killing weapon without anyone noticing and pulling the plug? Like I seriously don’t understand this argument.
>Can you explain how exactly an AI would develop a novel planet-killing weapon without anyone noticing and pulling the plug? Like I seriously don’t understand this argument.
That's the danger with creating a super intelligence, that you won't understand it. My dog is pretty smart, has good instincts and I'm quite confident he can survive in nature all by himself. But if a human decided to harm him, he doesn't stand a chance.
3 replies →
Seems like bioweapons (assuming "planet killing" means killing all the people, not literally blowing up Earth) are one way. Convincing the creator it would achieve their aim and could be contained when in fact it could not.
AI could invent something like social media. Obviously not social media because that's already been invented, but the next thing that slowly destroys society while seeming to be good.
> probably destroy almost anyone who survived the initial blast and fallout.
The difference between killing almost everyone, and killing everyone, is similar to the difference between almost getting pregnant and getting pregnant. A hundred thousand years later, a world where almost everyone had been killed would be teeming with humans.
> Can you explain how exactly an AI would develop a novel planet-killing weapon
I already answered that question in my previous comment, and the answer is that I am not going to explain how to develop weapons, but the larger point here is that we already can't explain how exactly AIs do innumerable things we see them doing every day, which is why we're having this conversation in the first place.
> I seriously don’t understand this argument
That is a point that I think everyone can agree on: you do not.
I think they mean we don’t have a deathstar. We can make the atoms here shitty. We can’t annihilate them or scatter them out of their own gravity well.
Life in theory could restart with our weapons, but what’s the point with the other scenario.
11 replies →
There are a couple different questions
1) Does AI just guide developers to speed up the pretty much the same data-train-test model that we've had? Or does it inspire new, better algorithms that inspire further, better algorithms until you have superintelligence able to run on the hardware of today. The first option doesn't to promise exponential increases whereas the second one seems to. Notably, neither the statistics nor the anecdotes we read here point to the second option.
2) Is super-intelligence something that - in control of people or out of control of people - poses a threat to humanity? That's a huge question but it seems like something deserving thought today. Still, everything indicates the things will unleashed without very consideration despite this, given current momentum. So, IDK, get ready.
I mean yeah, the main bottleneck in ML research used to be fixing python virtualenv hell when every existing torch repo breaks after two months of package updates, plus cuda and other version incompatibilities. That was like 95% of the work. Now it's more like 2% of the work and it's actually possible to get shit done without losing your mind.
Haha exactly. I don’t think people appreciate how slow AI research was to start with. It was absolute drudgery.