Comment by PeterStuer
11 days ago
So:
1. Encourage fencing off everything by default to maximize need for bypass
2. Offer bypass through payment
3. Profit!
You wouldn't believe the number of public administrations with public information that have (mostly unwittingly) had some lazy contractor put Cloudflare in front of their entire site, blocking even their RSS feeds from M2M. Yes, you can send them mails and call and sometimes, if they even understand the problem, they will fix it after a few months just before the next cheapest contractor is hired and we start all over again.
Not saying Cloudflare is just an extortion racket, but it's getting closer by the day.
I don't trust Cloudflare but this is not a problem they created. They solved a real problem with DDoS protection in the beginning and now AI crawlers increasing server cost is not a negligible problem. The CEO of iFixit called out Anthropic publicly for hitting their site a million times in 24 hours to scrape it. We are passed the point of good faith action from these AI companies. They are adversarial and need to be treated as such.
But they do create this problem. They could have specifically defaulted to not blanketing RSS feeds and other M2M specific pages. Instead, we are now in a situation where even daring to look at robots.txt can flag you as a bot.
True. For all those people who want to make excuses for Cloudflare, it's an excellent reminder that they've known about this problem for years and they still haven't fixed it.
Are they inept? Or do they really only care about things that bring them profit and that normalize their marginalization of non-paying groups? Which explanation makes the most sense?