Comment by Borealid
16 hours ago
I think there is something missed in how you're thinking about this problem.
Let's imagine that everything you say is 100% correct. LLMs can't access materials for developing weapons without weapons tests conducted at their behest, and humans choose not to - of their own volition - use world-ending weapons.
Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion.
A "superintelligence" persuades humans to perform weapons tests and then to give it the ability to use the weapon.
I think everything you've said is reasonable except the bit where you think step one of an "AI" exterminating humanity wouldn't look like a human doing it. In my opinion, the most likely vector for disaster is a human being who is "guided" by a computer model to do something unfortunate, not the model doing so directly.
What makes you think that persuasion isn't capped at some lower level than world-ending? There's a lot of incentive for ads to be as persuasive as possible, but they are so far from ending the world that it'd be science fiction for an ad to be that effective.
If you're just talking about convincing a few individuals to do experiments, then there are plenty of people who fall for scams everyday.
I think intuitively, intelligence is in some form about being able to predict the reaction that follows an action.
The extreme of intelligence is being able to create perfect statistical distributions for the effect of an intervention in the world. If you had that ability, you could turn it on the question of "what would this human choose to do if I were to give them instruction X" for any X.
I don't present any support for or evidence of:
- LLMs gaining intelligence by whatever means
- Extreme intelligence being possible
- Extreme intelligence being feasible
But I don't think I really need to provide support for the idea that something superintelligent would also be able to predict the actions of people under pressure in the same way someone of moderate intelligence can predict the actions of fluids under pressure.
You seem to describe humanity and AI's ability to predict and manipulate humanity a bit like how a physicist describes the ability of an apparatus to affect a given dynamical system. So let me put it like this: (1) You cannot model the effect f_predict(X) of your probe X with arbitrarily low |f_actual(X) - f_predict(X)|, and (2) you cannot guarantee the impact |f_actual(X)| is arbitrarily large (or that f_actual(X) saturates its codomain) when X in our case is restricted to inputs of the form, "tokens chosen in an LLM conversation."
11 replies →
> What makes you think that persuasion isn't capped at some lower level than world-ending?
People can be persuaded to kill themselves (see koolaid). People can be persuaded that arbitrary others are evil and must be exterminated (see Hitler). People can be convinced that dying for the cause is worth it (see 9/11 or kamikaze). People can believe that bringing about the end of the world would cause them to be saved into heaven (see Christian Zionism).
Why do you think it would be hard to convince any of these people to do something really stupid?
In fact, people don't even need to want to cause destruction to do something dumb - "I'm trying to save the world with free bio-fuel, help me create this bacteria" ("and I'm very smart, smarter than Elon Musk") - boom, pandemic.
It takes only one person to fall for it.
> People can be persuaded that arbitrary others are evil and must be exterminated (see Hitler)
If you are suggesting Hitler was persuaded that the Jews were evil, I don’t think that’s correct, he grew up in an environment where many people did believe that but they did not need to be persuaded, just like they did not have to be persuaded into their religion, culture etc. They just accepted the idea without much thought, which does not equate persuasion.
If you are saying the Allies were persuaded that Hitler was evil and must die, I think that may be closer to the truth since they knew the cost of invading Germany would be astronomical, potentially costing their own destruction, and they still stopped only when Hitler was taken, dead or alive. Some serious persuasion was needed for that.
2 replies →
And yet AI can't write decent landing page copy, or even a convincing cover letter. Seeing the obtuse "reasoning" from Claude daily, I'm very un-worried about AI being a world-ending persuasion engine.
1 reply →
> Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion.
Definitely not true as you can't use reinforcement learning to increase that capability. Only LLM capabilities that you can improve in an RL loop grow substantially.
Agreed. Like lots of this AI speculation appears to require the belief that AI will figure out how to build supervised training environments without more data.
I think this is very unlikely, but world changing if true.
Additionally, what does super intelligence even mean and why would we expect it to be obtainable from the data that we have?
I'd be delighted if someone can provide some definitions for these terms.
> Definitely not true as you can't use reinforcement learning to increase that capability.
What? Citation needed.
Well don’t you need a bunch of humans to test your persuasion capabilities? How would you run improvement tests in a loop??
2 replies →
>persuasion
Is there evidence that such soft skills are improving over time? I take as evidence to the contrary the fact that LLM writing has not substantially improved. I don't see that its powers of persuasion has either, whether that's through making strong arguments for a course of action or writing persuasive essays.
Only the labs knows the truth, but the question as to how many Americans used ChatGPT/Grok/Claude to help them decide how to vote in the most recent election is hopefully going to be something that historians will be able to unearth later.
I wonder if we’re approaching a cultural moment where people in general are doing exactly what they did when Google first became ubiquitous—search to confirm their biases—but now with LLMs instead of Google. But where Google would happily feed them some link to potential mis/disinformation, the AI is more likely to say “no, that’s not accurate, and here’s why…”
I doubt the number is low. Knowing everything I know about llms, I still fall into the same groove as I would if I was talking to a buddy online. That part is not far fetched.
I remember reading about research on using LLM's for political persuasion. It worked well.
There was also research on using LLM's in reddit to persuade, it also worked well.
I went to a talk by an author of one of these papers (David Rand). IIRC the format of the paper's experiment was collecting people with conspiracy theory beliefs and having LLMs with different strategies try to reduce their confidence in the conspiracy theory.
The best strategy, perhaps reassuringly, was tailoring facts to the person's worldview. I don't mean massaging facts, I mean that if the person thinks 9/11 was an inside job because of something about melting steel, then the LLM will present some sourced facts about how that actually works. And then it will do the same thing for all six of the sub-beliefs that support their overall belief. This was key to the strategy: the LLM has enough breadth and recall to exactly and patiently rebut most mistakes a person is making without making them defensive.
This strategy handily beat other things like flattery, selectively lying, etc. So while LLMs can clearly persuade, the evidence suggests it is easiest for them to persuade belief in something that is actually true.
This was in 2024 or 2025. Things might be different now.
I'm a physicist, and I will do my own calculations before I buy some solar/batteries in the near future, but I've been using ChatGPT and Claude to come up with potential candidates and ideas/scenarios. There's approximately 0% chance that they are not advertising what their owners are being paid for.
What’s your reasoning?
1 reply →
IMHO, one of the top-p vectors is probably something similar to what we've already witnessed with HuggingFace, (e.g. as part of an eval or RL run for cyber; things like Daybreak and Mythos are product lines), just worse in scope and somehow eluding monitoring again.
Not even that much persuasion is necessary.
We have the medium of persuasion: it's called money, and people do ALL SORTS of horrible things that they wouldn't otherwise do when incentivized by money.
We also already have the mechanism of a machine incentivizing and directing people with money, and that's called a corporation (the original paperclip maximizers, by the way).
So you just need an LLM CEO of a legal corporation with financial resources -> near unlimited possibility of real-world impacts over time.
> Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion. A "superintelligence" persuades humans to perform weapons tests and then to give it the ability to use the weapon.
Is there any evidence, either by way of quantifiable data or by step-by-step thought experiment, that suggests this is possible?
I consider it supremely silly, for example, that there is a graph of destructiveness as a function of persuasiveness where "y = manifest human extinction" is reached at some finite level of persuasiveness, as though persuasion is a single scalar that grows without bound, and the only safeguard between a conversation and WWIII is just a mind waiting for the right argument as expressed in a ChatGPT transcript. (Keep in mind, also, that you have already ceded to the OP that autonomous intellectual and physical self-improvement at singular trajectories is implausible.)
> Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion.
That feels like a bait and switch slyly introducing a whole new dimension. The discussion started with establishing that LLMs are extremely helpful with AI research. What is the evidence to justify rapidly improving persuasion skills? Even the scenario you've outlined hinges on a human being stupid enough to take LLM advice wholesale -- and every conversation that hypes up AI takes us one step closer to a world where some susceptible human feels amenable to that.
No bait-and-switch here. The discussion started with the idea that LLMs are helpful at improving LLMs. The goal of improving an LLM is to make the LLM more "intelligent" (whatever humans think that means). If they're helpful at that, they have to keep becoming more "intelligent".
I didn't present any evidence that more-intelligent means more-persuasive. If you don't think it does, then there is nothing to think about in the point I made, as no matter how "smart" the models get they will likely remain subject to human wills.
LLMs aren't magic, there's nothing they can do to persuade people to do weapons testing, it's an absurd scenario.
It feels like I don’t understand your definition of “persuade”, because the notion that people cannot be persuaded of things via reports and flashy animations is to me, absurd.
For what value of N do you think there exists a string of N characters I could have responded to your comment with that would have infected you with a stably self-replicating and capability-augmenting memetic virus with human extinction as its latent goal?
If not you, in this situation, with N characters delivered via a HackerNews comment, then who, in what situation, with what media and how much of it?
5 replies →
Do we have reason to believe they would be successful at other extreme feats of persuasion such as persuading a financial institution to transfer all of its holdings to an account controlled by the AI or persuading the Pope to renounce God?
If Putin can persuade/bribe/blackmail people to fly drones around European airports, then why wouldn't an LLM be able to do similar things?
Putin's persuasiveness does not derive from his intelligence.
The human tendency to want to finish things in a fraction of a second will definitely take us there.
It would be a human guiding the ai to guide other humans to end the world.
Not necessarily. If you haven't experienced an AI doing something you not only didn't want, but had expressly asked it not to do, you haven't used AI much.
That's a human problem, not an AI problem.
well it’s a human problem enabled by advanced AI.
Yes, basically a vastly more difficult and dangerous version of not letting young children play with matches, which will hopefully teach us many object lessons about safety and security long before it acquires launch codes or the means to use them.