Comment by raincole
15 hours ago
It's still very unclear to me how Anubis is supposed to work in the long term or even the medium term. The assumption is that scrappers, a.k.a. AI companies, a.k.a. those who hog all the RAM, are short of memory...?
15 hours ago
It's still very unclear to me how Anubis is supposed to work in the long term or even the medium term. The assumption is that scrappers, a.k.a. AI companies, a.k.a. those who hog all the RAM, are short of memory...?
The assumption is that requiring 1 second of compute before allowing a page load will have more impact on the people requesting 1000 pages/sec than it will on consumers requesting 1 page every minute.
Personally I don’t understand where this confusion comes from - it’s a simple economic tactic. Yes the large scale scrapers won’t run out of compute but putting anubis in front of a website demonstrably reduces the amount of crawling and that is the entire goal.
> requiring 1 second of compute before allowing a page load
After 1 solve, you get a cookie and can load tons of pages with it. Scrapers and user agents alike will need to spend the compute only once to get a valid session for themselves
Seems to be working so far though. Fwiw, lots of silly things stop bots until someone puts their mind (or tokens) to bypassing it, and then the blocking mechanism has to adapt. We'll see how it plays out, for now it's a lot better than begging big brother (cloudflare, google, or amazon captcha) for access, so I'm quite appreciative of what Techaro is doing
The issue is that scrapers are rotating their IP and essentially performing a DDoS attack. Anubis is part of a defense-in-depth solution. If scrapers reuse a cookie, traditional anti-abuse mechanisms will work.
9 replies →
https://people.kernel.org/monsieuricon/creepy-crawlies reports anubis effectiveness falling off
1 reply →
If you change your IP address, it invalidates the cookie. If you don't change your IP address, you can be blocked by IP address.
It’s not seconds though, it’s in the order of tens of milliseconds of work, it’s faster to complete the challenge than most http requests will take.
last difficulty 5 nonce 645376 in 9 ms (101.6MH/s, avx512-x16)
This takes many minutes to solve on a cellphone, 9 milliseconds on a cheap VPS. It’s not stopping anyone.
Yep, on my aging phone it takes quite w noticable amount of time to go through. But on my more recent laptop wnd workstations, it's a quick flash of a manga lolita.
1 reply →
>It’s not stopping anyone.
You could just spend 30 seconds googling instead of confidently asserting nonsense. There are plenty of people demonstrating that Anubis reduces bot traffic.
6 replies →
>The assumption is that requiring 1 second of compute before allowing a page load will have more impact on the people requesting 1000 pages/sec than it will on consumers requesting 1 page every minute.
Which doesn't hold. A 1s delay, despite being inperceptible is still a cost to human users. After all, 3600 x 1s delay adds up to an hour. Worse yet, the typical human visitor will always have to solve a challenge, because chances are it's the first time they're visiting your blog or whatever. Someone scraping reuse a valid session for at least a few more pages.
There's also the problem of time. It might be tempting to think the cost of a challenge is 1M cpu cycles or whatever, but the cost is better modeled as two parts, a time cost (for the human) and a cpu cost (for the computer). A scraper might have to pay the cpu cost, but not the time cost, because they can have their scraper solve challenges in their sleep. Unless the user is utterly destitute, their time is worth far more than whatever the compute cost is.
The assumption is that there will always be some population distribution in terms of effort expended as well as in terms of problems caused. At present it's the lowest effort actors that are causing the highest number of problems. So an 80% (or even just a 20%) solution largely solves the practical problem faced by site operators.
If against all odds it turns out that skiddies (or agents or AI billionaires or what have you) are willing to burn inordinate amounts of RAM and CPU just to scrape autogenerated cgit pages that are of approximately zero value to them to begin with then I guess we'll just have to cross that bridge when we come to it.
Okay but it's like no RAM at all. Thats the problem: you've got stupid bots causing a bunch of issues - these can be stopped.
And then regular users: who now can't really browse your site.
And then smart bots - anyone with even a moderate amount of funding in the AI space, who aren't even slightly slowed down.
The solution basically makes the favored customer well funded AI scrapers, not users.
Regular users store the cookie and do the challenge once. Bots do it over and over again throwing out the cookies so they're not rate-limited.
Extreme configurations with high difficulty do make for a frustrating experience on very old devices, but in practice it rarely gets in the way.
Only the tail end of what one could consider normal users (those running custom browsers or having JS disabled or nonfunctional) run into issues here, but they'll have these issues with any other bot-blocking mechanism that still offers some protection these days.
Apple's "ask your vendor for tokens to exchange for access" web standard gets around this, assuming those bespoke and Javascript-less browsers implement PATs, but that standard seems to be completely dead in the water outside of Safari.
> And then regular users: who now can't really browse your site.
That's entirely up to how the site operator configures the PoW challenge.
> anyone with even a moderate amount of funding in the AI space, who aren't even slightly slowed down
Do you have any evidence that it is the large well funded AI companies DoSing sites by (for example) scraping absurd numbers of permutations of cgit url parameters?
Even if it is (which I doubt), if all this accomplishes is getting them to spend 10 minutes optimizing their bots to not do that it will be a major win.
[dead]