Comment by harshreality
2 months ago
> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
It's massively less annoying than a captcha, which is both a longer delay (typically, at present) and a massive cognitive distraction/roadblock.
The anubis author has stated they recognize it's an arms race, but PoW scales. Captchas and other signals are already at the end of the road; any additional difficulty increases false bot-positives, which are already unacceptably high.
For websites running dynamic languages, a binary (anubis is in go) sentry that operates before[1] the website is forced to expend any resources, is usually a large improvement over a site-hosted captcha. I would rather, and I think most humans would agree, have to wait a few seconds, maybe even closer to a minute in the future, to get a website access token good for a day or a week, than be forced to solve a captcha.
The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
[1] this is true regardless of whether anubis is in reverse proxy mode or auth mode.
Proof of work does not scale. It trades something fungible and incredibly cheap (CPU) for something incredibly expensive (user-visible latency). There is no set of parameters where the cost is going to be a meaningful deterrent to any kind of abuse (even something as low-yield as scraping) without adding crippling amounts of latency to real users.
> The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
There is no dilemma. They get a token, they maybe do some automated multi-armed bandit per-site to figure out how to maximize the extraction rate they get from a single token, and then they use an IP for that many requests / that amount of time before ditching it.
The success of the ongoing Anubis rollout proves the opposite. People are used to slowly-loading websites - the rise of garbage SPAs has seen to that. Staring at a spinner for a second every once in a while is not an issue for genuine users.
On the other hand, the additional CPU usage rises the compute cost of scraping by several orders of magnitude. If you don't have to scrape this specific website, you'd be stupid not to move on and hit someone else.
Ideally the general cost of scraping would be high enough that it isn't affordable any more - especially anonymized - but considering the amount of money brainlessly being pumped into AI I doubt that'll happen any time soon.
You could of course also make the argument that the user's time is worth something as well and should be included as part of the cost, but that ship sailed a loooong time ago. If you care about that, you should be calling for the death of client-side Javascript and any form of advertising.
Scrapers generally aren't looking for random websites to scrape. They have a specific URL in mind. Only if the goal was DDoS would they not care which URL was accessed.
1 reply →
> It trades something fungible and incredibly cheap (CPU)
it could be RAM-bound, which is very much NOT cheap nowadays :)
It doesn't make the economics any different. In a browser environment, you're maybe looking at the acceptable lease being 100MB for 1 second. Much more than that, and you start hitting limits of what browsers will let you do on low-end phones. Longer than that, and we're back to the user-observable latencies being too long.
100MB for 1 second just is not much of a deterrent.
Yes, but the people with the RAM nowadays are the data centers, not the end users.
2 replies →
Anubis is by far the least annoying throttler I encounter. Entirely agreed, just crank it up when you get a flood, I much prefer waiting a couple seconds to interacting with custom UI for tens of seconds.
I'm so glad to see that (essentially) HashCash is coming back. Now we just need it for email, like it was originally designed for...
I looked this up and realised it’s the page I’d seen briefly on a range of websites lately. It’s not annoyed me at all. Not nearly as much as having to complete captchas with slow refreshing tiles.
A fun fact about Google captchas: they've often decided whether you will succeed or fail the captcha before you do the captcha.
9 replies →
I've usually been more annoyed at the surprise dissonance of "was that an anime girl on kernel.org?" than annoyed at the delay.
1 reply →
For me Cloudflare is worse, it takes more than 5 seconds, where as anubius take 1-2 secs.
funny with all the IP information they have, cloudflare cannot do a better job. (I am on IPv6)
and most of the time, its on marketing product pages like in framework main site, which can be cached.
Imo the worst is recaptcha. At least with cloudflare the work you have to provide is minimal. With recaptcha it can take me much longer than 5 seconds, and lately I have trouble even completing their challenges correctly. Nowadays if I see a (recaptcha) captcha I drop the site unless I must visit it for some reason, it is not worth the time, the effort or the annoyance.
6 replies →
> just crank it up when you get a flood,
A few months ago there was a story posted here about someone who completely eliminated crawlers on their website with Anubis.
I think it was getting upvoted before users were clicking the article because if you did, you had to leave the Anubis PoW page open for several minutes before you could get into the site. The Anubis difficulty scale is unintuitive and the difference between a small delay and becoming unusable is easy to cross.
From my understanding this is also how cloudflare bot protection has worked for a long time, and then they look for entropy in user input to confirm the user is human. Also how recaptcha without images works.
Google and Cloudflare both are not just looking at entropy of mouse movements, that was cracked years ago, they are fingerprinting you and correlating your session with all your activity cross domains to score your botlike behavior.
8 replies →
Supposedly, but not really. I regularly encounter sites where cloudflare serves me with an ambiguous ban notice rather than a proof of work. What's worse is that these apparent IP bans take effect even if I already had a valid active session (ie previously passed the check).
Yes, a VPN involved. That doesn't make it okay and notice that anubis by default works without issue (though possibly with a more difficult challenge) in the exact same scenario.
4 replies →
Cloudflare often just straight up blocks me or makes me do a captcha. IMO those are both much worse than Anubis
2 replies →
Also by doing network traffic analysis
Except when it throws you into a reload loop. It's pretty buggy, and trivial to bypass.
And contrary to grandparent, PoW only worked because it was a novel thing to work around, a simple "type human" prompt would've worked as well.
When anubis gets widespread enough users will still run the PoW in javascript or whatever while the scrapers will run much more optimized native code, so no, it doesn't scale.
Putting aside the question of whether it will continue to work, even somewhat, against botnets, I find your first paragraph confusing.
Reload loops, or being able to "bypass" anubis (unless you merely mean bypassing it for the token validity period by solving a challenge), sound like misconfigurations. There's no reason for anubis itself to cause reload loops; it's tricky to configure a webserver to use it in some scenarios.
Any ability to bypass anubis probably means the site is using it in auth/challenge mode only, and then misconfigured their webserver's auth checking. Or it's a bug. If you mean the double-spend tavis mentioned in his blog post which previously made the HN frontpage, that was patched right after it was reported to the maintainer almost a year ago.
Or you can go full Reddit and just block anything that seems even remotely suspicious.
Your sibling, roommate, neighbor that uses your internet, previous IP owner, posts too much? You get blocked too.
Using VPN? Blocked.
Your iPhone is too old, blocked.
Your screen brightness too low? Believe or not, blocked.
My forum got scraped so hard that the ISP blackholed the IPv4 several times a week.
I've ended up putting only IPv6 on the domain. It's running this way for 2 years already.
It's fun to log in with a banned Reddit account on a shared IP, which bans every other Reddit login on the same IP.
Even worse, not blocked, shadowbanned.
Parks and Rec reference?
I quit reddit because of all of this nonsense. After 15 years on reddit, my life has been much better for quitting it. Reddit is a cesspool.
> Your screen brightness too low? Believe or not, blocked.
... What?!
PoW barely affects the "residential proxies" aka. malware botfarms. The IPs are free for them and siphoning additional system resources for PoW doesn't matter at all for them. PoW only affects the large centralised scraping by the AI providers, which are not operating behind "residential proxies".
Most users of residential proxies just get a SOCKS5 address and routing, they don't actually get computational resources of the infected systems beyond that. The user of the proxies, the operator of what the article describes as a control node, would be the device responsible for the PoW.
Do you have any evidence that AI providers aren't using residential proxies?
Yes, the status quo right now is merely bandwidth... but if there was money to be made in providing a small amount of compute for the sake of solving the next gen captcha's, you can bet offerings will expand to meet it.
It's impossible to prove a negative. They could all be running secondary scrapers using malware proxies... but what we do have is plenty of evidence of them using fixed IP pools with appropriate user agents. I don't see any Chinese user agents... so two guesses who may be driving the bulk of these AI scraper requests via residential proxies.
Residential proxy bandwidth is extremely expensive, comparatively speaking. It can be up to $1 per GB but is more typically about $0.20 per GB.
Right, this is orthagonal to the discussion though. While the IPs and bandwidth might be free, managing the malware botnet and trying to keep a low profile so as to not attract attention of authorities, makes it a risky market to cater to.
Those rates are still cheaper than some datacentres charge in parts of the world.
If you pop my machine and use it to route 100 MBit/s, I might not notice for months.
If I hear the fan spinning at night, you're probably getting caught immediately.
If you pop my mom's TV box and use it to route data within the connection's capabilities, you're getting away with it. If you consume a little bit of resources, still. If you consume enough to be useful for these kind of challenges, chances are her TV playback will start to stutter, which will be resolved by taking the compromised TV box, and removing the malware using advanced mechanical means called "a trash compactor".
If you're tech savvy enough to monitor your equipment, you're probably smart enough not to install "free VPNs" or "free stream pirate apps" to get pwned in the first place. You probably changed your router's default password and kept its firmware up to date, and don't give your TV internet access, right? That's more than Mom has done.
>chances are her TV playback will start to stutter
video decoding is hardware accelerated, and there's probably enough excess compute to be able to do some sort of PoW challenge. Besides, unlike humans, bots aren't in a hurry, so they can spread out the work across a long time to minimize disruption.
the device acts as a proxy. i don't think any browser is running on the device, it is just forwarding packets.
> The anubis author has stated they recognize it's an arms race, but PoW scales.
The scraper wars are largely between script kiddies and people with both deep intimate networking and DOM knowledge. Yes greyhairs, I’m looking at you.
The problem is, you can’t PoW every page load and resource request because the user experience will suck and people will run away. And that window - the gap between what people will tolerate vs draconian enforcement - is exactly what the scrapers exploit.
And looking at the PoW options out there - I’ve seen at least one PoW WAF (honestly can’t remember if azure or amazon) have their PoW boil down to repeated trigonometric functions, ie very optimisable.
It’s a neat concept, but the answer and future to my eyes look bleak.
Oh, but you can PoW every page.
Your typical end user doesn't switch IPs that often, so it's fine to Anubis them again when they do. A scraper, on the other hand, has a tradeoff to make between rotating ips often (requiring a challenge on every request) or keeping only a few IPs (making cross-request identification much more valuable and reliable).
> Oh, but you can PoW every page.
They meant you can’t PoW every page transition.
If clicking every link on your website throws you back to another Anubis page for 2-3 seconds, users will bounce.
That’s why Anubis does an up front challenge and then you’re good for a while. It’s a really low cost for the scrapers.
2 replies →
Anubis's default 1-week token lifetime may not be nearly enough to dissuade enough scraper networks to make a difference, particularly with the default weight->difficulty level hierarchy, but that's for individual site admins to determine.
We can all argue based on how we envision "ideal" scraper networks being run and whether the web-PoW concept would stand up to that. However, what matters at present is that anubis helps many sites cope with misbehaving bot scrapers written by the script kiddies you mention, who don't care if the internet burns as long as they finish their scrape 1 hour faster. If anubis motivates them to devote a few brain cells to make their scrapers smarter, they may also fix the scrapers to not take down the sites they're scraping.
One of the ideas behind Anubis was to incentivize a scraper to stop hiding, because every change of identity brings another challenge page.
I don't think PoW scales, because if the bot authors get serious they'll start using native implementations that are much more efficient than the web ones real users are running. In theory maybe Anubis could start using WebGPU to help close that gap, but then anyone without WebGPU support is out of luck.
Then again, a large portion of the problem seems to be bots making way too many requests and in general not being optimized in the first place, and this does help filter those out.
If that happens the browser engines (all what, 3 of them?) can add a PoW API to call into native code. Or a pathologically scalar algorithm can be adopted so that wasm is good enough. RandomX or something close to it probably qualifies.
There are PoW approaches that even the playing field between data centers and desktops. RandomX is my favorite.
For what it's worth I'm working on hashx support. It's just going to take a bit to ship while I do browser testing with broken browser configs.
Interesting. How do they tell the difference between legitimate and forged ip owner records?
7 replies →
At least anubis works for me. (I run umatrix)
Unfortunately whatever HN is using routinely blocks my login with "Sorry."
some websites just always give me 403.
> Unfortunately whatever HN is using routinely blocks my login with "Sorry."
I believe that's the HN application itself, not a WAF in front of it.
HN is surprisingly very very guilty of a whole lot of anti-user patterns and behaviour that other companies get regularly lamented.
Poor accessibility, bad mobile support, no options to delete content beyond a narrow window.
13 replies →
[dead]
Well, we don't use a captcha either. If it were a choice between a captcha and a proof of work system, we'd have to reevaluate things. Luckily, for now, we're able to get away with a much lighter touch.
> but PoW scales
Not if the honest party is doing it in a browser: The same computer can so any POW so much faster in C than any amount jf JS and WASM that it will never ever ever be a contest.
> becoming much more obvious and easy to block, or they have to use massive amounts of compute.
If you believe this, please contact me: I think compute is free[1] and can probably help you out.
[1]: https://news.ycombinator.com/item?id=30175269
Can you not design a PoW that is most efficient in a browser? Don't brute force hashes like Hashcash/Bitcoin, do something similar to RandomX instead but in JS. Browsers ought to run the fastest JS interpreters already so if interpreting JS becomes the bulk of the work, that attack might not work. Maybe even involve the DOM or whatever else makes sense.
> Browsers ought to run the fastest JS interpreters already
Well they don't.
Users want the website to work sooner, and care little about whether a for-loop of elements take 10ms or 20ms if it only happens once.
JS can be AOT compiled if you can wait a few _seconds_ -- which users don't want, so browsers don't bother.
Our attacker however, rightly observes they only have to pay that compilation cost once.
2 replies →
[flagged]
> The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
You can't do that any more. Too many ISPs, especially mobile carriers, don't hand out anything resembling a fixed IP address any more. It's CGNAT and constantly changing IP addresses alllll the time now.
On a small enough site (even LWN might qualify) the chance of two random sets of client IPs intersecting can be quite low.
Private trackers do this. If they ban a user that geolocates to a certain city and ISP, they'll ban new signups from that city and ISP because there's probably only a few users from the same city and ISP. And then report to their friends at other trackers, that a user with that city and ISP is trying to evade a ban.
Anubis appears to be a temporarily-useful stopgap that has been cargo culted into prominence and an expectation of permanent usefulness, for reasons I don't fully understand.
The cost of solving the default Anubis PoW is negligible on cloud servers, and it's even lower if you use native code rather than JavaScript to solve it, which Tavis Ormandy helpfully demonstrated last year (https://lock.cmpxchg8b.com/anubis.html). If Anubis were to be even more widely adopted, botnet operators would surely adopt and optimize native code solvers en masse.
So Anubis doesn't do much to stop bots, but it makes otherwise lightweight websites (little JavaScript or interactivity) almost unusable on low-resource systems like my old phone or an old Atom-based nettop.
> when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape
This "IP-bound proof-of-work" thing is gonna kill multipath TCP and bring down IPv6 with it. Uffff.
> If Anubis were to be even more widely adopted, botnet operators would surely adopt and optimize native code solvers en masse.
Then anubis adopts it itself, increases the amount of work that needs to be done and the bar stays the same again for everyone? Seems like mostly a non-issue unless there is an arms race towards ever more optimized solvers which I don't believe is possible.
> > If Anubis were to be even more widely adopted, botnet operators would surely adopt and optimize native code solvers en masse.
> Then anubis adopts it itself, increases the amount of work that needs to be done and the bar stays the same again for everyone?
No, the bar doesn't "stay the same" for everyone interacting with Anubis.
My otherwise-perfectly-usable 8-year-old phone, which can't be patched to run a native solver, becomes even more unusable on sites gates with proof-of-work challenges like Anubis.
This is the whole problem with PoW. It forces thousands or millions or billions of client devices to do increasing amounts of useless work which is relatively easy for cloud-based attackers to adapt to, but very difficult for hardware-constrained and software-ossified mobile clients to adapt to.
In other words, it asymmetrically punishes the clients that it's not intending to punish.
I'm not gonna wait a minute to read an article. Instead, I'll either just leave, or go query it from archive.$tld that bypasses it for me.
[flagged]
PoW can theoretically scale effectively infinite because it can mine cryptocurrency. Millions of compromised IoT devices hitting your server? Now you have enough money for a faster server.
It doesn’t matter that the challenge must be verified: present multiple challenges, some are verified while others mine crypto.
This is called “installing a cryptominer on your web page” and is generally considered illegitimate.
> generally considered illegitimate
But why? Obviously an unjustified cryptominer is bad, like unnecessarily slow JavaScript, but this one has a good purpose and to the user is no different than PoW.