Comment by uludag
19 days ago
I'm always curious how poisoning attacks could work. Like, suppose that you were able to get enough human users to produce poisoned content. This poisoned content would be human written and not just garbage, and would contain flawed reasoning, misjudgments, lapses of reasoning, unrealistic premises, etc.
Like, I've asked ChatGPT certain questions where I know the online sources are limited and it would seem that from a few datapoints it can come up with a coherent answer. Imagine attacks where people would publish code misusing libraries. With certain libraries you could easily outnumber real data with poisoned data.
Unless a substantial portion of the internet starts serving poisoned content to bots, that won’t solve the bandwidth problem. And even if a substantial portion of the internet would start poisoning, bots would likely just shift to disguising themselves so they can’t be identified as bots anymore. Which according to the article they already do now when they are being blocked.
>even if a substantial portion of the internet would start poisoning, bots would likely just shift to disguising themselves so they can’t be identified as bots anymore.
Good questions to ask would be:
- How do they disguise themselves?
- What fundamental features do bots have that distinguish them from real users?
- Can we use poisoning in conjunction with traditional methods like a good IP block lists to remove the low hanging fruits?
(I was going to post "run a bot motel" as a topline, but I get tired of sounding like broken record.)
To generate garbage data I've had good success using Markov Chains in the past. These days I think I'd try an LLM and turning up the "heat".
Wouldn't your own LLM be overkill? Ideally one would generate decoy junk more much efficiently than these abusive/hostile attackers can steal it.
I still think this could worthwhile though for these reasons.
- One "quality" poisoned document may be able to do more damage - Many crawlers will be getting this poison, so this multiplies the effect by a lot - The cost of generation seems to be much below market value at the moment
I didn't run the text generator in real time (that would defeat the point of shifting cost to the adversary, wouldn't it?). I created and cached a corpus, and then selectively made small edits (primarily URL rewriting) on the way out.
Reddit is already full of these...
Sorry but you’re assuming that “real” content is devoid of flawed reasoning, misjudgments, etc?