Comment by fc417fc802
2 days ago
Please consider installing one of the many PoW schemes such as anubis rather than use these cloudflare "features". I increasingly encounter outright blocks rather than any sort of captcha when visiting cloudflare "protected" sites. Each individual site isn't particularly important to me but it's depressing to watch the process unfold like this. You really are choosing to erode the core basis of the internet if you go this route.
PoW schemes like Anubis don't work. Increasingly bots are using headless browsers and are basically able to solve captchas, proof-of-work(s) and basically bypass all any any attempts to block them. It's becoming impossible to stop bots from hammering your sites/services for unwanted traffic.
There's one or a small number of actors who are DDoSing the internet right now, and those are the ones you care about. What's the purpose for blocking non-DDoS bots?
Seems like there is no POW "solving" if the POW in question is literally solving some sort of mathematical puzzle. It has a bounds and we can just scale the bounds depending on how much abuse we are seeing.
I've never understood why that isn't more common honestly. It would be cool if I could actually transfer POW from one domain to another as well. Probably wouldn't stop scrapers, but it would likely stop low margin fraud type things.
The most recent captchas that involve basic reasoning can block majority of bots today.
Name one, that an LLM can't solve. The only real mechanism that continues to still work (but doesn't scale) is "Human in the loop".
You are asserting that they don't work without explanation or evidence. Meanwhile it isn't clear why they wouldn't and indeed they appear to accomplish the stated goal of severely rate limiting scrapers.
IIRC there was a blog post here on HN recently explaining in detail why Anubis doesn't keep out AI bots.
1 reply →
> You are asserting that they don't work without explanation or evidence.
Oh, no, that isn't how it works — you made a claim first (albeit indirectly), with no evidence. It's on you
2 replies →
Surely I don't need to explain how headless browsers work, bot proxies and distributed crawlers to you do I? If you haven't experienced this first-hand, I can understand. Maybe it warrants a blog post. But I can assure you, these don't work at scale, they might only stop the "less sophisticated" bots and maybe (just maybe) some unwanted spam.
1 reply →
>I increasingly encounter outright blocks rather than any sort of captcha when visiting cloudflare "protected" sites.
???
Unless you're browsing around with the googlebot user agent string, you should be getting turnstile challanges at most, not blocks. And if you're getting a turnstile challenge it's unclear how it's different than an anubis challenge. If you're outright blocked, it's probably a site decision (eg. block all VPNs or block everyone not from a given country) rather than cloudflare's.
Try using Firefox with "resist fingerprinting" enabled, on OpenBSD, if you want an idea of how insidious Cloudflare's increasing prevalence is for the openness of the web. Inexplicable 403 responses for entire domains for no apparent reason, no "captcha", just blocked outright. Heck, it happens on Linux too, and even without "resist fingerprinting" enabled.
Happens with older chrome browsers as well. Last supported windows 7 version. Started this month.
> And if you're getting a turnstile challenge it's unclear how it's different than an anubis challenge.
You are guaranteed to pass an Anubis challenge eventually [0], whereas it's possible to get stuck forever in an infinitely-looping Turnstile challenge.
> If you're outright blocked, it's probably a site decision (eg. block all VPNs or block everyone not from a given country) rather than cloudflare's.
Cloudflare blocks legitimate users itself sometimes [1].
[0]: Unless you run into a bug, but Anubis is open source, so you can always submit a patch upstream. I've done this myself, and I can confirm that it's relatively straightforward.
[1]: https://news.ycombinator.com/item?id=43329320
>You are guaranteed to pass an Anubis challenge eventually [0]
That's a double edged sword because bots will eventually get through too, and unlike humans, their time is dirt cheap.
>Cloudflare blocks legitimate users itself sometimes [1].
I never ran into this issue despite using seemingly maximally suspicious configs like tor browser. I can't say the same for some other vendors.
5 replies →
I’ve been blocked by Turnstile without possibility of challenge, no way to access nor know why. And it was with standard iOS and Windows machines and no VPN or anything. It happens.
Yesterday Cloudflare told me I was a bot and couldn't access this.
yeah, googlebot user agent thing was a thing maybe around 10 years ago but I don't think it's working today. Maybe if you start the http session from gcloud, it might still work.
Please don't use Anubis, it makes visiting websites very difficult (often multiple minutes wait times) on old hardware and low-end smartphones.
As things stand right now, the current web may not be possible to maintain for old and low-end phones given the costs imposed by AI training crawlers.
Please propose an alternative to both Cloudflare and Anubis, that shields websites against inhuman traffic without frequent operator intervention (or otherwise negates the capacity costs they pay for AI crawling) and is compatible with low-end smartphones.
Certainly, I imagine Anubis would be interested in adopting it if it’s effective!
The solution is building simpler and more efficient websites. You won't win the arms race, so make it irrelevant. Any random VPS hosting a static website can already serve millions of requests an hour.
Great you now made the internet unusable for half the planet.
3 replies →
Personally I think that's a better outcome than the alternative.
However another option is to support both. Have a challenge page that requires the visitor to select one of several options. Cloudflare could be one of those.
What's the alternative? We're stuck, either cloudflare or anubis
what kind of hardware would take minutes to complete the checks?
It really depends on the difficulty level chosen by the website. I'm testing on both a Xiaomi Redmi 5 and my university library's computer.
> Please consider installing one of the many PoW schemes such as anubis
Why not go all the way and mine monero instead of just completely wasting the work?
because then someone will complain that you're stealing their CPU cycles or something, on the website they chose to visit and execute
"Stealing" CPU cycles is exactly what PoW bot protection is doing. The whole point is to add cost to the bots so they decide it's too expensive and give up.
The only difference is the cycles are getting converted into heat now. They could be getting converted into monero instead. It's still heat but at least creator got some money for it.
Not sure why Anubis is getting so much hype on HN, but honestly, it is not the solution. A real solution would use behavioral modeling. Most browser fingerprinting issues are already largely solved anyway.
As someone who is regularly blocked by CF and G - no, they are not. Even worse, they should not be, because that would mean the complete end of privacy online (not that we are far from that). Anubis works because at scale it wastes bots' resources (time mainly).
I think you’re confusing bots that crawl a website with bots used to launch DDoS attacks.
it works because the global ddos adversary doesn't run javascript
I'm genuinely baffled that this is your lived experience. I wonder if you only look at a very restricted set of websites?
PoW is a reasonable solution as a fallback when other metrics flag a client.
How? First, they can solve the Anubis challenge with native code, so they can solve them faster than genuine users. Second, the cost is nothing compared to training LLMs, plus they will just move the work to the residential proxies that they have access to (so, someone is paying through their TV's electricity bill).
Anubis only work(s|ed) great for a while when crawlers were not prepared for these challenges. Security through obscurity.
2 replies →
Adding the link to GitHub here if anyone is curious:
https://github.com/techaroHQ/anubis