← Back to context

Comment by jeroenhd

7 hours ago

Shitty scrapers hitting images over and over again is also a problem. Not just /pic/1.webp - /pic/2.webp, but in practice I've seen three AI scrapers that would request things like /pic/1.webp?version=webp&foo=bar and other random query parameters that none of my domains even use. It's just stuck brute-forcing files or something, sucking up loads of bandwidth downloading the same image for no reason.

It also depends on if you want your content to be scraped. Any bot ignoring robots.txt should be considered potentially malicious and worth considering such mechanisms for.