Comment by iririririr
9 days ago
> The AI companies won't just scrape IA once, they're keeping come back to the same pages and scraping them over and over. Even if nothing has changed.
Maybe they vibecoded the crawlers. I wish I were joking.
9 days ago
> The AI companies won't just scrape IA once, they're keeping come back to the same pages and scraping them over and over. Even if nothing has changed.
Maybe they vibecoded the crawlers. I wish I were joking.
Isn't this just how crawlers work? How do you know if a page has changed if you don't keep visiting it?
HEAD requests
Scrapers could also use the "Cache-Control", "Expires" and "If-Modified-Since" headers properly, to reduce their traffic a little, but do they?