Comment by jmyeet
9 hours ago
It's been about 20 years since I read Neal Stephenson's Anathem and (SPOILERS) one thing I think about is the Reticulum, which is the Internet on this world. One aspect is that it's full of what we would now call slop. A lot of it is subtly changed by agents and there are other agents you can use (and, I assume purchase?) that will weed out the fake information. It's basically an arms race of disinformation and slop.
I've thought about this many times over the years. I think we've now reached the point where that's going to happen. And state actors seem like the obvious first offenders.
Google is relatively good at filtering out astroturfed pages from the top of search results but they've also have years when a Turkish government page denying the Armenian genocide was the top result for that event so it's a bit of a mixed bag. Google relies a lot on authority of a domain and that's been relatively effective.
It seems like this authority concept is missing from LLM training and a random comment on Reddit is treated with as much authority as an article by the New York Times. So it's now orders of magnitude easier to poison LLMs with Reddit comments and submissions. And for anyone who doesn't know, brigading Reddit is remarkably easy. It takes as little as a few hundred accounts to get something onto r/all if you're clever. Israeli hasbara operations are absolutely capable of this. And they engage in it.
I'm also reminded in a weird way of pre-atomic steel. For anyone who doesn't know, this is steel made before 1945 that is incredibly low in radiation because we hadn't detonated any nuclear bombs yet. It's valuable for making MRIs and the like that are super sensitive. We actually go and recover WW2 shipwrecks just for the steel. It is a finite resource.
Well, that's what pre-2022 written content is now. It's slop-free content to train AI models on because training them on AI slop is of much more limited value. This goes for everything from books to Youtube videos to Reddit posts. Just like pre-atomic steel, we now have pre-slop human content.
I honestly don't know where we go from here.
The comparison to pre-atomic steel is poignant. I have found myself preferring pre-AI sources in some cases even (especially?) for computer related information. It is difficult to view pre-slop info as a limited resource without implying decline until the status quo changes again.
I hate to break it to you. But it took war'ing state-like actors less than a week to start using LLMs for propaganda purposes. There are pictures that most of what some people think/thought were real that never happened. Pictures that form their opinion on political topics. It was literally instantaneous. It was also remarkably predictable in almost all ways...sigh. Even the topics were predictable. And most people predictably fell for it, even the majority of the posters on this very site.