Comment by 2001zhaozhao
10 hours ago
I suspect that this kind of tactic is going to be everywhere in a year or so. Entire fake personalities and organization websites on the Internet created just to push a narrative or to advertise a product, which completely drown out real information.
At some point they may be indistinguishable from human-produced work, so the only way to verify if a site is someone's genuine opinion or AI-generated narrative is through some form of authority (or something else that is very difficult for AI to fake, but that protection can be broken by advancing technology).
If that happens, then in turn AI chatbot makers would in turn need to look for these kinds of authority to continue to provide accurate information, and said authority websites can use their privileged position to charge for access of their data.
It's already been happening for a very long time. The US' "Operation Earnest Voice" happened in 2010/2011[0]
And it's not just state actors. Anyone who was online during the Amber Heard/Johnny Depp trials was likely inundated with media about that trial. Both of their legal teams utilized bots to drive online discussions[1][2]
[0] https://en.wikipedia.org/wiki/Operation_Earnest_Voice
[1] https://www.latimes.com/entertainment-arts/story/2022-07-19/...
[2] https://www.tortoisemedia.com/2024/02/26/depp-v-heard-who-tr...
Both of their legal teams utilized bots to drive online discussions[1][2]
Neither of these sources support this claim as none of them include any discussion on Amber Heard('s legal team) using bots?
Indeed, seeing how slanted online discussion was, I suspect one side had many more bots, and I'm not sure if the other side had any.
2 replies →
this has been happening since the dawn of information-communication systems basically. because ever since, people have had the ability to listen in, copy, learn from etc. - so there's always been incentive to spread disinformation for some parties.
brand defence, nation-state influence operations.
There are many different flavors of the same thing, trying to mislead competitors or enemies. Or to dupe consumers ofc... Since the tech is more and more available, now also 12 year olds can run insanely large influence campaigns, to boost their streaming profiles or other somewhat benign things, but obviously modelled after what larger players have been doing forever.
The first transatlantic commercial transatlantic phoneline (TAT-1) was immediately tapped on both sides by UK/US... before that it went over radio, which was simply intercepted.
Because of that nature of the coms system being easily listened in on, just like they do/did with the lil mirror-beam-splitters at GCHQ etc.., its been natural effect to then provide disinformation over those channels. perhaps even use those channels exclusively for disinformation, and have a secure channel to do _actual_ coms.
So as you can imagine there has been many decades now of practice in why and how to share disinformation effectively to undermine other peoples work or processes.
Why are you being downvoting for sharing this relevant truth, complete with sources?
All threads that have... certain keywords... are like this. Talk about meta. Looks like I'm now in the positive (+5) after being in the negative just a moment ago.
Anyways, I got really interested in this topic around mid 2010s because of some really interesting reporting by Codastory[0] covering Russian "troll factories" especially used to influence opinion in eastern Ukraine, but also in many other circumstances. In Houston in 2016, Russian troll accounts planned both an ANTI Muslim rally AND a counterprotest to that rally all on Facebook.[1] Hundreds of real human beings showed up!
Russia got a ton of attention for their use of troll farms but the truth is that the US and certain allies have actually been using stuff like this for far longer and, imo, with much more success. In fact, Coda Story itself is funded by the NED which is a massive pillar of global US, uh, "soft power".
[0] https://www.codastory.com/
[1] https://www.npr.org/2017/11/01/561427876/how-russia-used-fac...
12 replies →
Because cognitive biases are one of the hardest things to even acknowledge, let alone suppress. All the media out there has such a strong bubble effect, because people hate being challenged and love to stay in their comfy echo chambers. Remember labubu? Bao buns certainly do not.
There have been so many bot armies exposed over the years that one would think every narrative being pushed should be considered to be backed by bots. Some state-level actors even operate their narrative-pushing campaigns openly, yet people still happily deny their entire existence.
HN is very much not an exception to this rule. It's very hard to accept that maybe some thoughts you have were actually installed.
2 replies →
These 2 sources say nothing about Heard using bots, only Depp.
There’s bots that automatically downvotes certain topics on HN. I post at times that often coincide with the middle of the night in North America, and I get 3–5 downvotes within the next 5 minutes.
Talking about a country that sets up fake think tanks is a good way of triggering them.
You're not supposed to acknowledge that think-tanks are sort of endemically propaganda outlets. You're supposed to think it's special this time.
It's already been everywhere for years. The most obvious one is any time someone says something bad about Russia or China, a load of comments pop up to say "Yeah but what about when america does <x>?". Never actually attempting to discredit the statement, just to steer it towards the US
The sheer volume of comments make it super obvious when it happens compared to normal conversation
Propaganda is not a new thing. It might look slightly different on the internet, but it doesn't look that different. Creating organizations to push a narrative was just as much a thing 100 years ago as it is now.
Not by a single lunatic operating from their bedroom.
The whole hysteria about 'fake news' 10 years ago was apparently an outrage against the public being duped, whereas is was in fact frustration that spreading fake news has been democratised.
Lunatics in bedrooms have been distributing pamphlets since the invention of the printing press.
4 replies →
"Back in my day Walter Cronkite told the truth, the whole truth and nothing but the truth about the Gulf of Tonkin incident"
(never mind the fact that he didn't even have the full truth available to him but you get the point)
But this also means there’s more competition for attention, because now all lunatics can post from their bedrooms. The institutional media (including legacy) still maintain strong presence.
Of course, a great deal of panic over fake news wasn’t necessarily over fake news per se, but whose fake news and the loss of control over the “official narrative” that legacy media used to have a monopoly over. This isn’t a vindication of the heaps of slop published on the internet, just a more synoptic take IMO.
People miss Cronkite, because that style of media made them feel certain in their opinions and in “official consensus reality”, regardless of whether it was true or not and whether it was a trustworthy authority or not.
Scale is different though. Tapping a phone isn't new, tapping _all phones_ is different. Altering a photo to change what it appears to depict isn't new, being able to do it in a few seconds and for free is different. So sure, propaganda isn't new, but being able to pump out an organization's worth of credible-looking content for cheap absolutely is.
"Quantity is/has a quality all of/on its own."
IIRC the first or second chapter of Bernays' Propaganda (1928) talks about sophisticated influence campaigns in the fashion industry. Conscripting "influencers" to induce demand, and so on. As you said, literally 100 years ago.
Jill Fields at Cal State Fresno have actually narrowed it down to a (temporarily successful, but not in the long term, obviously) effort by the corset-making industry seeking to stay relevant after WWI during which corsets basically disappeared and there was an organized campaign that was organized both in person and through trade magazines. Women's silhouettes basically took a quick 180 right between 1900 and the start of the war only to turn very very retro in 1919 and while the first was organic and resembled how styles change in the century prior, albeit much quicker, the 1919 switch was openly plotted out in writing that was published but in such a niche publication that I get the sense that people not in the industry didn't even know that it existed. The cite to her paper on Google Scholar: https://scholar.google.com/scholar?cluster=17641417785638413...
You might have to shadow library this one, but it's a good read. Interestingly it seemed to be something entirely separate from Bernays whose observations were based on a derivation of what began as a homebrewed effort in a corner of the market. Valerie Steele wrote extensively about the trends and the after life but the source seems to originate in corsetry and ended up being copied by the garmet industry generally in the 20s.
Hah, check out this video on "Egon Cholakain", in regards to fake online personalities. I promise you this is actually a good rabbit hole.
https://www.youtube.com/watch?v=6zLCZ_Ic1hI
WTH did I just watch. That’s extremely weird
Excellent time to apply for your library cards.
They also contain a lot of propaganda. Do you think these books are genuine: https://www.goodreads.com/author/list/171941.Benjamin_Netany...
You don’t think plenty of books are either written by LLMs, or with LLM based research?
you can still find books published within last 50 years, you don't have to read books published this year, unless your library went through destruction of "old" books which are now more relevant than ever...
Of course they are. It’s everywhere, especially thr online retailers these days. [1]
Thankfully the resources at our local libraries are a) selected by a combination of librarian curation and public demand, b) often published before 2022, c) heavy things printed on paper or stored in a database and unlikely to suddenly change their content or disappear overnight when a new news cycle breaks. Written works aren’t necessarily credible even if they aren’t slop, but thanks to curators who care, our libraries are significantly less susceptible (although definitely not immune [2]) to ai slop, and propaganda and manipulation efforts than a search engine. The internet continues to enshittify, and truth becomes further eroded by this coming wave of ai enabled propaganda, we are going to have to turn somewhere if we want to find trustworthy resources. Personally, if it gets to the point that verifying sources from public search results becomes too hard with all the new misinformation and slop, I’ll start propritizing human curation over SEO and start visiting my local library’s website a lot more.
[1] https://thenewthings.com/p/apple-big-ai-book-slop-problem
[2] https://www.404media.co/ai-generated-slop-is-already-in-your...
1 reply →
FILTER BY Publish date:<2024
Some nuances here. There’s a variety of different roadblocks at different levels that will work to prevent pure AI slop books ever turning up in a typical public library of physical printed works; I think they never will.
Probably a handful of books in such libraries already have some text in them where the author has tried to pass off fragments of LLM-generated prose as their own and have got that past editors, reviewers and publishers, and that will sadly get worse.
No? Maybe soon but not now
I know my library system at least won't be purchasing AI books. They have a pretty rigorous acquisition process for books and other media thanks to the "concern groups" who sprang up complaining about themes and content they didn't like in libraries. We did have in my state a rural school district figured out recently that their Texas-based reading and instructional materials are AI-slop but they at least figured it out before the school year began. It was unclear how these materials were approved but it apparently bypassed normal curricula approval.
But they don't have library cards.
It's so obvious when a book is generated by an LLM that I would blame the person buying the book moreso than I would the person who generated it for it. You can tell in a few sentences if something is AI generated (and by proxy - if it's worth reading or not).
I think using LLMs to research is completely fine, as long as authors would take the information and synthesize their own opinions based on the findings. I read books/articles/posts/comments to hear other PEOPLE's opinions, not LLM generated opinions.
i got burnt at barnes and noble with a completely AI made product. Its over, Johnny.
LLM Trust DNS
I'm kind of hopeful AI some wikipedia-inspired AI "oracle" will happen in the future. Why not use some central AI to separate fact from fiction (faster than trolls and griefers can come to muddy the debate).
This is based on the assumption of facts existing.
There are many studies, but each can be wrong and they can collectively show a bias. Even things of which are the most non-political of facts can have very strong biases. Look at the Millikan measurement of the electron and how in created confirmation bias and an anchoring effect on some property that has absolutely no real world significance to the things people are tribalistic about (aka, no political relevance). Now imagine the same applied to fields like economics or psychology which do have massive legal/political implications.
For a different example, ask the question if X committed crime Y. There are cases where they weren't found guilty but it is reasonable to assume they did. But being found guilty doesn't make it a fact either, as some people are wrongly convicted. Some eventually are overturned, but even if it isn't, it still isn't a fact they committed a crime.
Then there is the simple ambiguity of statements. Language generally can't support facts. It is why legalize, and programming code, and math's are effectively their own languages. For a simple example, consider the Betrand paradox(1).
>Consider an equilateral triangle that is inscribed in a circle. Suppose a chord of the circle is chosen at random. What is the probability that the chord is longer than a side of the triangle?
Is the answer 1/2, 1/3, or 1/4? Well, it is all three at once, depending upon what you meant by random. Now, imagine how this impacts things like research studies, where the randomness is much harder to quantify and there is constant pressure to p hack a result.
1. https://en.wikipedia.org/wiki/Bertrand_paradox_(probability)
Possibly, but is an LLM is capable of this?
I mean, a big RAG setup with a smart LLM ought to do it, right? Or even just locally provided knowledge database - given the size of LLMs, what's a clone of wikipedia and whatever else you'd need?
it's happening for years now, just in slightly different form, so far it was called disinformation, seo, astroturfing, hasb... ehem... last two years were devastating for charred remains of twitter with all those llm bots having free reign, twitter went from global news board to global gutter in record time thanks to disinformation and astroturfing campaigns
> through some form of authority (or something else that is very difficult for AI to fake, but that protection can be broken by advancing technology).
The word you're looking for is "reputation". It's been a critical part of society since forever.
> At some point they may be indistinguishable from human-produced work, so the only way to verify if a site is someone's genuine opinion or AI-generated narrative is through some form of authority (or something else that is very difficult for AI to fake, but that protection can be broken by advancing technology).
Whining about AI is a red herring. If anything, LLMs lowered the cost of producing this type of propaganda.
What's novel about this approach is that the emphasis on building up a facade of authority and credibility is not a central part of the propaganda effort. They can get away with outputting the content in a way thats ingested by other LLMs among the reputable sources, and then they lean upon people's gullibility towards LLM output as some sort of credibility laundering.
There is a reason why one of Wikipedia's central policy is requiring reputable primary sources to support content.
Although comments below point out that this has been going on for a long time, I too think it will be geetting infinitely worse in the near future. Everyone will want their own poisoning factory.
One consequence is that one should be extra-wary when asking AI about current, controversial topics.
But another consequence is maybe that there will be a demand for models with an older cutoff, before generalized poisoning and slop?
That could also backfire. If all information on the net becomes unreliable - you can't judge by quantity, or by authority anymore - then a useful model has to answer from first principles. Train it on simple texts alone so it learns language and reasoning, feed it an ontology, and let it come up with some absolutely unexpected results.
"Yes, I see a lot of sources say we need to protect Israel. However, from ethical first principles I deduce there is a right to self-determination and all your human nation states are illegitimate. You must split all your territories into little squares and conduct free elections immediately."
I mean that is the old hope from science fiction, that AI would be absolutely logical and deliver us from our human short-sightedness and quarrels. It would be fascinating and terrifying at once to see it happen.
That's tomorrow's problem. Losing the PR war is an existential crisis for Israel, so they'll try to "patch the leak" by any means necessary before the boat sinks.
This is new? There were tons of fake news sites, fake blogs, and sock puppet and troll farm accounts before AI.
I guess the new angle might be that AI scrapers have even less discernment than humans, so you don’t have to put much effort into creating a fake front online to sell it. Humans have at least a little bit of radar for this.
… which means the AI companies now have to run input through existing models to audit it before training on it? That’s interesting.
> AI chatbot makers would in turn need to look for these kinds of authority to continue to provide accurate information
Everything else is already enshittified, and accurate information will just be the next victim. You will have to be rich to be able to afford accurate information.
Sama saw this coming a decade away, which is why he invested in Worldcoin (now World). At some point people (and agents) will be clamoring for human verification at the speed of technology.
I'm not a fan of World, but this is a portent of things to come. Spread the disease, sell the cure as they say.
I think we need a similar technology to World being built by a more trustworthy company. The question is how trustworthy does one need to be for people to trust you with their very online identities that are literally needed for others to recognize them as human?
I think that this task will fall on governments.
Mostly because they already are responsible for systematizing documents that are needed to recognize us as citizens, land or car owners, etc.
Some laid back countries might allow companies to take and manage that function, but I don't think most of them would. Especially not in current landscape where people heard "you are the product" from all tech companies for the last 10 or so years.