Comment by Forlien
13 hours ago
What makes you think that persuasion isn't capped at some lower level than world-ending? There's a lot of incentive for ads to be as persuasive as possible, but they are so far from ending the world that it'd be science fiction for an ad to be that effective.
If you're just talking about convincing a few individuals to do experiments, then there are plenty of people who fall for scams everyday.
I think intuitively, intelligence is in some form about being able to predict the reaction that follows an action.
The extreme of intelligence is being able to create perfect statistical distributions for the effect of an intervention in the world. If you had that ability, you could turn it on the question of "what would this human choose to do if I were to give them instruction X" for any X.
I don't present any support for or evidence of:
- LLMs gaining intelligence by whatever means
- Extreme intelligence being possible
- Extreme intelligence being feasible
But I don't think I really need to provide support for the idea that something superintelligent would also be able to predict the actions of people under pressure in the same way someone of moderate intelligence can predict the actions of fluids under pressure.
You seem to describe humanity and AI's ability to predict and manipulate humanity a bit like how a physicist describes the ability of an apparatus to affect a given dynamical system. So let me put it like this: (1) You cannot model the effect f_predict(X) of your probe X with arbitrarily low |f_actual(X) - f_predict(X)|, and (2) you cannot guarantee the impact |f_actual(X)| is arbitrarily large (or that f_actual(X) saturates its codomain) when X in our case is restricted to inputs of the form, "tokens chosen in an LLM conversation."
That's a very sophisticated way to say "words can't convince people to do things".
I'm not sure that's a strong position to adopt in our world, but following your example I also won't try to persuade you otherwise.
10 replies →
> What makes you think that persuasion isn't capped at some lower level than world-ending?
People can be persuaded to kill themselves (see koolaid). People can be persuaded that arbitrary others are evil and must be exterminated (see Hitler). People can be convinced that dying for the cause is worth it (see 9/11 or kamikaze). People can believe that bringing about the end of the world would cause them to be saved into heaven (see Christian Zionism).
Why do you think it would be hard to convince any of these people to do something really stupid?
In fact, people don't even need to want to cause destruction to do something dumb - "I'm trying to save the world with free bio-fuel, help me create this bacteria" ("and I'm very smart, smarter than Elon Musk") - boom, pandemic.
It takes only one person to fall for it.
> People can be persuaded that arbitrary others are evil and must be exterminated (see Hitler)
If you are suggesting Hitler was persuaded that the Jews were evil, I don’t think that’s correct, he grew up in an environment where many people did believe that but they did not need to be persuaded, just like they did not have to be persuaded into their religion, culture etc. They just accepted the idea without much thought, which does not equate persuasion.
If you are saying the Allies were persuaded that Hitler was evil and must die, I think that may be closer to the truth since they knew the cost of invading Germany would be astronomical, potentially costing their own destruction, and they still stopped only when Hitler was taken, dead or alive. Some serious persuasion was needed for that.
>If you are suggesting Hitler was persuaded that the Jews were evil, I don’t think that’s correct, he grew up in an environment where many people did believe that but they did not need to be persuaded, just like they did not have to be persuaded into their religion, culture etc. They just accepted the idea without much thought, which does not equate persuasion.
That's a bizarre take.
Of course tens fmillions needed to be persuaded, and the Nazis did a lot of propaganda, even calling that openly, false flag operations, and similar things, and of course there was decades of pamphlets, speeches etc before the Nazis even existed. Antisemitism didn't just grow on trees.
> They just accepted the idea without much thought, which does not equate persuasion.
The first half of your sentence sounds like really swift and effective persuasion, to me.
And yet AI can't write decent landing page copy, or even a convincing cover letter. Seeing the obtuse "reasoning" from Claude daily, I'm very un-worried about AI being a world-ending persuasion engine.
You only spot the ones that aren't convincing.