Comment by Roark66
13 hours ago
Cloudflare is ridiculous. I can't even open it using Cromite (privacy enhanced, but not over the top, android browser).
I get:
blog.adafruit.com Verifying you are human. This may take a few seconds.
blog.adafruit.com needs to review the security of your connection before proceeding.
And this hangs forever. What difference does it make if I access this site using a browser (blocked anyway) or I asked my LLM to fetch the content? I bet my LLM coukd get it anyway as I'm using basic local scraping with firecrawl for backup. So my LLM if it fails to retrieve using my basic local crawl4ai will use my paid firecrawl api and those guys can scrape EVERYTHING.
I do not understand why do you (as a site owner) care? Are these bots generating so much traffic? Can you set it up to serve text only version to them then?
this isn’t an adafruit-specific stance, it’s a web-wide problem. automated scraping and bot traffic is enough to take independent sites offline, and cloudflare is a tool we use to keep the site available at all. we publish full-text rss with no blockers here: https://blog.adafruit.com/rss . if cloudflare trips on your browser and you want an article, email me and i’ll send it in whatever format you want, we're always working to make it easier, it's hard, would rather have help than snarks and dunks.
My office uses ZScalar and lots of sites automatically block that because the company running the product make the product seem like an "open anonymous proxy".