Comment by brookst
19 days ago
Assuming there is at least one already linked somewhere on the web, the crawlers already have logic to handle these.
19 days ago
Assuming there is at least one already linked somewhere on the web, the crawlers already have logic to handle these.
if you can detect them, maybe feed them low iq stuff from a small llama. add latency to waste their time.
It would cost you more than it costs them. And there is enough low IQ stuff from humans that they already do tons of data cleaning.
> And there is enough low IQ stuff from humans that they already do tons of data cleaning
Whatever cleaning they do is not effective, simply because it cannot scale with the sheer volumes if data they ingest. I had an LLM authoritatively give an incorrect answer, and when I followed up to the source, it was from a fanfic page.
Everyone ITT who's being told to give up because its hopeless to defend against AI scrapers - you're being propagandized, I won't speculate on why - but clearly this is an arms race with no clear winner yet. Defenders are free to use LLM to generate chaff.