Comment by solid_fuel
13 hours ago
The assumption is that requiring 1 second of compute before allowing a page load will have more impact on the people requesting 1000 pages/sec than it will on consumers requesting 1 page every minute.
Personally I don’t understand where this confusion comes from - it’s a simple economic tactic. Yes the large scale scrapers won’t run out of compute but putting anubis in front of a website demonstrably reduces the amount of crawling and that is the entire goal.
> requiring 1 second of compute before allowing a page load
After 1 solve, you get a cookie and can load tons of pages with it. Scrapers and user agents alike will need to spend the compute only once to get a valid session for themselves
Seems to be working so far though. Fwiw, lots of silly things stop bots until someone puts their mind (or tokens) to bypassing it, and then the blocking mechanism has to adapt. We'll see how it plays out, for now it's a lot better than begging big brother (cloudflare, google, or amazon captcha) for access, so I'm quite appreciative of what Techaro is doing
The issue is that scrapers are rotating their IP and essentially performing a DDoS attack. Anubis is part of a defense-in-depth solution. If scrapers reuse a cookie, traditional anti-abuse mechanisms will work.
>If scrapers reuse a cookie, traditional anti-abuse mechanisms will work.
Scrapers can't change their strategy? It's not hard to think of mitigations: keep on using a session until you're banned, and failing that rotate your IP. Nowadays you don't even need to think. You can probably tell a LLM "write me a scraper that has adaptive anti-anti-bot mechanism".
8 replies →
https://people.kernel.org/monsieuricon/creepy-crawlies reports anubis effectiveness falling off
I would disagree with the Anubis effectiveness falling off, but the llm bros are throwing life itself into training to defeat any “perceived” challenge.
If you change your IP address, it invalidates the cookie. If you don't change your IP address, you can be blocked by IP address.
It’s not seconds though, it’s in the order of tens of milliseconds of work, it’s faster to complete the challenge than most http requests will take.
last difficulty 5 nonce 645376 in 9 ms (101.6MH/s, avx512-x16)
This takes many minutes to solve on a cellphone, 9 milliseconds on a cheap VPS. It’s not stopping anyone.
Yep, on my aging phone it takes quite w noticable amount of time to go through. But on my more recent laptop wnd workstations, it's a quick flash of a manga lolita.
[flagged]
>It’s not stopping anyone.
You could just spend 30 seconds googling instead of confidently asserting nonsense. There are plenty of people demonstrating that Anubis reduces bot traffic.
That’s because its weird and mostly irrelevant, it’s nothing to do with the proof of work being any sort of economic impediment. It takes Claude a few minutes to write a near instant solver, that’s the maximum proof of work cost this adds.
1 reply →
>There are plenty of people demonstrating that Anubis reduces bot traffic.
"Anubis reduces bot traffic" and "[the proof of work] is not stopping anyone" isn't contradictory, especially if anubis isn't being benchmarked against non-POW alternatives like IP/ASN bans, or tls fingerprinting.
3 replies →
>The assumption is that requiring 1 second of compute before allowing a page load will have more impact on the people requesting 1000 pages/sec than it will on consumers requesting 1 page every minute.
Which doesn't hold. A 1s delay, despite being inperceptible is still a cost to human users. After all, 3600 x 1s delay adds up to an hour. Worse yet, the typical human visitor will always have to solve a challenge, because chances are it's the first time they're visiting your blog or whatever. Someone scraping reuse a valid session for at least a few more pages.
There's also the problem of time. It might be tempting to think the cost of a challenge is 1M cpu cycles or whatever, but the cost is better modeled as two parts, a time cost (for the human) and a cpu cost (for the computer). A scraper might have to pay the cpu cost, but not the time cost, because they can have their scraper solve challenges in their sleep. Unless the user is utterly destitute, their time is worth far more than whatever the compute cost is.