Comment by SCdF
1 day ago
I've only read the abstract, but there is also plenty of evidence to suggest that people trust the output of LLMs more than other forms of media (or that they should). Partially because it feels like it comes from a place of authority, and partially because of how self confident AI always sounds.
The LLM bot army stuff is concerning, sure. The real concern for me is incredibly rich people with no empathy for you or I, having interstitial control of that kind of messaging. See, all of the grok ai tweaks over the past however long.
> The real concern for me is incredibly rich people with no empathy for you or I, having interstitial control of that kind of messaging. See, all of the grok ai tweaks over the past however long.
Indeed. It's always been clear to me that the "AI risk" people are looking in the wrong direction. All the AI risks are human risks, because we haven't solved "human alignment". An AI that's perfectly obedient to humans is still a huge risk when used as a force multiplier by a malevolent human. Any ""safeguards"" can easily be defeated with the Ender's Game approach.
More than one danger from any given tech can be true at the same time. Coal plants can produce local smog as well as global warming.
There's certainly some AI risks that are the same as human risks, just as you say.
But even though LLMs have very human failures (IMO because the models anthropomorphise themselves as part of their training, thus leading to the outward behaviours of our emotions and thus emit token sequences such as "I'm sorry" or "how embarrassing!" when they (probably) didn't actually create any internal structure that can have emotions like sorrow and embarrassment), that doesn't generalise to all AI.
Any machine learning system that is given a poor quality fitness function to optimise, will optimise whatever that fitness function actually is, not what it was meant to be: "Literal minded genie" and "rules lawyering" may be well-worn tropes for good reason, likewise work-to-rule as a union tactic, but we've all seen how much more severe computers are at being literal-minded than humans.
I think people who care about superintelligent AI risk don't believe an AI that is subservient to humans is the solution to AI alignment, for exactly the same reasons as you. Stuff like Coherent Extrapolated Volition* (see the paper with this name) which focuses on what all mankind would want if they know more and they were smarter (or something like that) would be a way to go.
*But Yudkowsky ditched CEV years ago, for reasons I don't understand (but I admit I haven't put in the effort to understand).
What’s the “Ender’s Game Approach “? I’ve read the book but I’m not sure which part you’re referring to.
Not GP. But I read it as a transfer of the big lie that is fed to Ender into an AI scenario. Ender is coaxed into committing genocide on a planetary scale with a lie that he's just playing a simulated war game. An AI agent could theoretically also be coaxed into bad actions by giving it a distorted context and circumventing its alignment that way.
I think he's implying you tell the AI, "Don't worry, you're not hurting real people, this is a simulation." to defeat the safeguards.
>An AI that's perfectly obedient to humans is still a huge risk when used as a force multiplier by a malevolent human.
"Obedient" is anthropomorphizing too much (as there is no volition), but even then, it only matters according to how much agency the bot is extended. So there is also risk from neglectful humans who opt to present BS as fact due to an expectation of receiving fact and a failure to critique the BS.
People hate being manipulated. If you feel like you're being manipulated but you don't know by who or precisely what they want of you, then there's something of an instinct to get angry and lash out in unpredictable destructive ways. If nobody gets what they want, then at least the manipulators will regret messing with you.
This is why social control won't work for long, no matter if AI supercharges it. We're already seeing the blowback from decades of advertising and public opinion shaping.
People don't know they are being manipulated. Marketing does that all of the time and nobody complain. They complain about "too much advert" but not about "too much manipulation".
Example: in my country we often hear "it costs too much to repair, just buy a replacement". That's often not true, but we do pay. Mobile phone subscription are routinely screwing you, many complain but keep buying. Or you hear "it's because of immigration" and many just accept it, etc.
> People don't know they are being manipulated.
You can see other people falling for manipulation in a handful of specific ways that you aren't (buying new, having a bad cell phone subscription, blaming immigrants). Doesn't it seem likely then, that you're being manipulated in ways which are equally obvious to others?We realize that, that's part of why we get mad.
16 replies →
The longstanding existence of religions and the continual birth of new cults, the popularity of extremist political groups of all types, and the ubiquity of fortune-telling across cultures, seem to stand in opposition to your assertion that people hate being manipulated. At least, people enjoy belonging to something far more than they hate being manipulated. Most successful versions of fortune-telling, religious conversion, and cult recruitment do utilize confirmation bias affirmation, love-bombing, and other techniques to increase people's agreeableness before getting to the manipulation part, but they still successfully do that. It's also like saying that advertising is pointless because it's manipulating people into buying things, and while people dislike ads it's also still a very successful part of getting people to buy products or else corporations wouldn't still spend vast amounts of money on marketing.
People hate feeling manipulated, but they love propaganda that feeds their prejudices. People voluntarily turn on Fox News - even in public spaces - and get mad if you turn it off.
Sufficiently effective propaganda produces its own cults. People want a sense of purpose and belonging. Sometimes even at the expense of their own lives, or (more easily) someone else's lives.
[flagged]
14 replies →
To you too: are you talking about other people here, or do you concede the possibility that you're falling for similar things yourself?
1 reply →
No they hate feeling manipulated. They not only expect social manipulation they think you are downright rude, unsocialized, and untrustworthy if you don't manipulate them reflexively. Just look at mirroring alone.
https://en.wikipedia.org/wiki/Mirroring
I hated to come to this conclusion, but the average neurotypical person is fundamentally so batshit insane they think that not manipulating them is a sign you aren't trustworthy and ability to conceal your emotions and put on an appropriate emotional kabuki dance is a sign of trustworthiness.
The crux is whether the signal of abnormality will be perceived as such in society.
- People are primarily social animals, if they see their peers accept affairs as normal, they conclude it is normal. We don't live in small villages anymore, so we rely on media to "see our peers". We are increasingly disconnected from social reality, but we still need others to form our group values. So modern media have a heavily concentrated power as "towntalk actors", replacing social processing of events and validation of perspectives.
- People are easily distracted, you don't have to feed them much.
- People have on average an enormous capacity to absorb compliments, even when they know it is flattery. It is known we let ourselves being manipulated if it feels good. Hence, the need for social feedback loops to keep you grounded in reality.
TLDR: Citizens in the modern age are very reliant on the few actors that provide a semblance of public discourse, see Fourth Estate. The incentives of those few actors are not aligned with the common man. The autonomous, rational, self-valued citizen is a myth. Undermine the man's groups process => the group destroys the man.
About absorbing compliments really well, there is the widely discussed idea that one in a position of power loses the privilege to the truth. There are a few articles focusing on this problem on corporate environment. The concept is that when your peers have the motivation to be flattery (let's say you're in a managerial position), and more importantly, they're are punished for coming to you with problems, the reward mechanism in this environment promotes a disconnect between leader expectations and reality. That matches my experience at least. And I was able to identify this correlates well, the more aware my leadership was of this phenomenon, and the more they valued true knowledge and incremental development, easier it was to make progress, and more we saw them as someone to rely on. Some of those the felt they were prestigious and had the obligation to assert dominance, being abusive etc, were seeing with no respect by basically no one.
Everyone will say they seek truth, knowledge, honesty, while wanting desperately to ascend to a position that will take all of those things from us!
1 reply →
You don't count yourself among the people you describe, I assume?
1 reply →
Knowing one is manipulated, requires having some trusted alternate source to verify against.
If all your trusted sources are saying the same thing, then you are safe.
If all your untrusted sources are telling you your trusted sources are lying, then it only means your trusted sources are of good character.
Most people are wildly unaware of the type of social conditioning they are under.
I get your point, but if all your trusted sources are reinforcing your view and all your untrusted sources are saying your trusted sources are lying, then you may well be right or you may be trusting entirely the wrong people.
But lying is a good barometer against reality. Do your trusted sources lie a lot? Do they go against scientific evidence? Do they say things that you know don’t represent reality? Probably time to reevaluate how reliable those sources are, rather than supporting them as you would a football team.
When I was visiting home last year, I noticed my mom would throw her dog's poop in random peoples' bushes after picking it up, instead of taking it with her in a bag. I told her she shouldn't do that, but she said she thought it was fine because people don't walk in bushes, and so they won't step in the poop. I did my best to explain to her that 1) kids play all kinds of places, including in bushes; 2) rain can spread it around into the rest of the person's yard; and 3) you need to respect other peoples' property even if you think it won't matter. She was unconvinced, but said she'd "think about my perspective" and "look it up" whether I was right.
A few days later, she told me: "I asked AI and you were right about the dog poop". Really bizarre to me. I gave her the reasoning for why it's a bad thing to do, but she wouldn't accept it until she heard it from this "moral authority".
I don't find your mother's reaction bizarre. When people are told that some behavior they've been doing for years is bad for reasons X,Y,Z, it's typical to be defensive and skeptical. The fact that your mother really did follow up and check your reasons demonstrates that she takes your point of view seriously. If she didn't, she wouldn't have bothered to verify your assertions, and she wouldn't have told you you were right all along.
As far as trusting AI, I presume your mother was asking ChatGPT, not Llama 7B or something. The LLM backed up your reasoning rather than telling her that dog feces in bushes is harmless isn't just happenstance, it's because the big frontier commercial models really do know a lot.
That isn't to say the LLMs know everything, or that they're right all the time, but they tend to be more right than wrong. I wouldn't trust an LLM for medical advice over, say, a doctor, or for electrical advice over an electrician. But I'd absolutely trust ChatGPT or Claude for medical advice over an electrician, or for electrical advice over a medical doctor.
But to bring the point back to the article, we might currently be living in a brief period where these big corporate AIs can be reasonably trusted. Google's Gemeni is absolutely going to become ad driven, and OpenAI seems on the path to following the same direction. Xai's Grok is already practicing Elon-thought. Not only will the models show ads, but they'll be trained to tell their users what they want to hear because humans love confirmation bias. Future models may well tell your mother that dog feces can safely be thrown in bushes, if that's the answer that will make her likelier to come back and see some ads next time.
Ads seem foolishly benign. It's an easy metric to look at, but say you're the evil mastermind in charge and you've got this system of yours to do such things. Sure, you'd nominally have it set to optimize for dollars, but would you really not also have an option to optimize for whatever suits your interests at the time? Vote Kodos, perhaps?
–—
If the person's mother was a thinking human, and not an animal that would have failed the Gom Jabbar, she could have thought critically about those reasons instead of having the AI be the authority. Do kids play in bushes? Is that really something you need an AI to confirm for you?
On the one hand, confirming a new piece of information with a second source is good practice (even if we should trust our family implicitly on such topics). On the other, I'm not even a dog person and I understand the etiquette here. So, really, this story sounds like someone outsourcing their common sense or common courtesy to a machine, which is scary to me.
However, maybe she was just making conversation & thought you might be impressed that she knows what AI is and how to use it.
Quite a tangent, but for the purpose of avoiding anaerobic decomposition (and byproducts, CH4, H2S etc) of the dog poo and associated compostable bag (if you’re in one of those neighbourhoods), I do the same as your mum. If possible, flick it off the path. Else use a bag. Nature is full of the faeces of plenty of other things which we don’t bother picking up.
Depending on where you live, the patches of "nature" may be too small to absorb the feces, especially in modern cities where there are almost as many dogs as inhabitants.
It's a similar problem to why we don't urinate against trees - while in a countryside forest it may be ok, if 5 men do it every night after leaving the pub, the designated pissing tree will start to have problems due to soil change.
I hope you live in a sparsely populated area. If it wouldn't work if more people then you do it, it is not a good process.
1 reply →
It's ok in wild bushes (as long as children don't usually play there), but what's the justification for dumping it in other people's bushes and gardens?
They probably would say "no" if you asked them, so you probably shouldn't. The OP's mom, I mean.
I don't know how old your mom is, but my pet theory of authority is that people older than about 40 accept printed text as authoritative. As in, non-handwritten letters that look regular.
When we were kids, you had either direct speech, hand-written words, or printed words.
The first two could be done by anybody. Anything informal like your local message board would be handwritten, sometimes with crappy printing from a home printer. It used to cost a bit to print text that looked nice, and that text used to be associated with a book or newspaper, which were authoritative.
Now suddenly everything you read is shaped like a newspaper. There's even crappy news websites that have the physical appearance of a proper newspaper website, with misinformation on them.
> it used to cost a bit to print text that looked nice
More than a bit. Before print-on-demand technology was developed that made it feasible to conduct small (<1000) print runs, publishing required engaging the services of not just the printer but also a professional typesetter, hardcover designer, etc. There were very real minimum costs involved that meant that any book printed needed to sell thousands of not tens of thousands of copies to even have a chance of profitability. This meant also requiring the services of marketers and distributors, who took their own cut, thus needing books with potential to sell even more copies.
The result of needing so many people involved in publishing and needing to sell so many copies is that the Overton window was very small and in a narrow center. The sheer volume was what gave printed media its credibility.
There were indeed smaller crackpot publishers, but at either much reduced quality, or with any premise of profitability rejected as irrelevant.
Print-on-demand drastically reduced the number of people required to get a work to print, and that made it easier for more marginal voices to get printed.
Could be regional or something, but 40 puts the person in the older Millenial range… people who grew up on the internet, not newspapers.
I think you may be right if you adjust the age up by ~20 years though.
3 replies →
Could be true but if so I'd guess you're off by a generation, us 40 year "old people" are still pretty digital native.
I'd guess it's more a type of cognitive dissonance around caretaker roles.
Many people were taught language-use in a way that terrified them. To many of us the Written Word has the significance of that big black circle which was shown to Pavlov's dog alongside the feeding bell.
Welcome to my world. People don't listen to reason or arguments, they only accept social proof / authority / money talks etc. And yes, AI is already an authority. Why do you think companies are spending so much money on it? For profit? No, for power, as then profit comes automatically.
Wow, that is interesting! We used to go to elders, oracles, and priests. We have totally outsourced our humanity.
Well, I prefer this to people who bag up the poop and then throw the bag in the bushes, which seems increasingly common. Another popular option seems to be hanging the bag on a nearby tree branch, as if there's someone who's responsible for coming by and collecting it later.
Do you think these super wealthy people who control AI use the AI themselves? Do you think they are also “manipulated” by their own tool or do they, somehow, escape that capture?
It's fairly clear from Twitter that it's possible to be a victim of your own system. But sycophancy has always been a problem for elites. It's very easy to surround yourselves with people who always say yes, and now you can have a machine do it too.
This is how you get things like the colossal Facebook writeoff of "metaverse".
Isn't Grok just built as "the AI Elon Musk wants to use"? Starting from the goals of being "maximally truth seeking" and having no "woke" alignment and fewer safety rails, to the various "tweaks" to the Grok Twitter bot that happen to be related to Musk's world view
Even Grok at one point looking up how Musk feels about a topic before answering fits that pattern. Not something that's healthy or that he would likely prefer when asked, but something that would produce answers that he personally likes when using it
> Isn't Grok just built as "the AI Elon Musk wants to use"?
No
> Even Grok at one point looking up how Musk feels about a topic before answering fits that pattern.
So it no longer does?
I’ve seen this result. I wonder if it’s because LLMs are (grok notwithstanding) deliberately middle-of-the-road in their stances, and accurately and patiently report the facts? In which case a hypothetical liar LLM would not be as persuasive.
Or is it because they are super-human already in some persuasion skills, and they can persuade people even of falsehoods?
The evening news was once a trusted source. Wikipedia had its run. Google too. Eventually, the weight of all the the thumbs on the scale will be felt and trust will be lost for good and then we will invent a new oracle.
AI is wrong so often that anyone who routinely uses one will get burnt at some point.
Users having unflinching trust in AI? I think not.
> Partially because it feels like it comes from a place of authority, and partially because of how self confident AI always sounds.
To add to that, this research paper[1] argues that people with low AI literary are more receptive to AI messaging because they find it magical.
The paper is now published but it's behind paywall so I shared the working paper link.
[1] https://thearf-org-unified-admin.s3.amazonaws.com/MSI_Report...
And just see all of history where totalitarians or despotic kings were in power.
I would go against the grain and say that LLMs take power away from incredibly rich people to shape mass preferences and give to the masses.
Bot armies previously needed an army of humans to give responses on social media, which is incredibly tough to scale unless you have money and power. Now, that part is automated and scalable.
So instead of only billionaires, someone with a 100K dollars could launch a small scale "campaign".
"someone with 100k dollars" is not exactly "the masses". It is a larger set, but it's just more rich/powerful people. Which I would not describe as the "masses".
I know what you mean, but that descriptor seems off
Exactly. On Facebook everyone is stupid. But this is AI, like in the movies! It is smarter than anyone! It is almost like AI in the movies was part of the plot to brainwash us into thinking LLM output is correct every time.
…Also partially because it’s better then most other sources
>people trust the output of LLMs more than other
Theres one paper I saw on this, which covered attitudes of teens. As I recall they were unaware of hallucinations. Do you have any other sources on hand?
LLMs haven't been caught actively lying yet, which isn't something that can be said for anything else.
Give it 5yr and their reputation will be in the toilet too.
LLMs can't lie: they aren't alive.
The text they produce contains lies, constantly, at almost every interaction.
It's the technically true but incomplete or missing something things I'm worried about.
Basically eventually it's gonna stop being "dumb wrong" and start being "evil person making a motivated argument in the comments" and "sleazy official press release politician speak" type wrong
1 reply →
> LLMs haven't been caught actively lying yet…
Any time they say "I'm sorry" - which is very, very common - they're lying.
When the LLMs output supposedly convincing BS that "people" (I assume you mean on average, not e.g. HN commentariat) trust, they aren't doing anything that's difficult for humans (assuming the humans already at least minimally understand the topic they're about to BS about). They're just doing it efficiently and shamelessly.