I just found out about this when it came to the front page of Hacker News. I really wish I was given advanced notice. I haven't been able to put as much energy into Anubis as I've wanted because I've been incredibly overwhelmed by life and need to be able to afford to make this my full time job. Support contracts are being roadblocked, and I just wish I had the time and energy to focus on this without having to worry about being the single income for the household.
I don't think Xena is going to promote themselves but I looked at their GitHub sponsor's page. They have a monthly goal of $5k but haven't even hit $1.5k. Combined with their Pateron they get a whopping $2k/mo.
There's a lot of love for Anubis alone, but this doesn't translate to a third of what a junior dev makes. How do we expect to turn these highly beneficial open source products when people cannot pass off a small portion of their savings to those that make the tools.
I hope you figure it out Xena. Even though I'm only on the user side of things I've liked Anubis more than other solutions like cloudflare.
Thanks! I just wish I could afford to work on this full time, or at least even part time. It would help me a lot and prevent me from having to work what is effectively two full time jobs. Rent and food keep getting more expensive in Canada.
Good luck, I'm sorry for all of this speculation and people attacking your solution instead of suggesting concrete improvements to help fight the problem.
Thanks. It means a lot. Today has not been a good day for me. It will be fixed. Things will get better, but this has to rank up there in terms of the worst ways to find out about security issues. It sucks lol.
I actually don't understand who Anubis is supposed to "make sure you're not a bot". It seems to be more of a rate limiter than anything else. It self-describes:
> Anubis sits in the background and weighs the risk of incoming requests. If it asks a client to complete a challenge, no user interaction is required.
> Anubis uses a proof-of-work challenge to ensure that clients are using a modern browser and are able to calculate SHA-256 checksums. Anubis has a customizable difficulty for this proof-of-work challenge, but defaults to 5 leading zeroes.
When I go to Codeberg or any other site using it, I'm never asked to perform any kind of in-browser task. It just has my browser run some JavaScript to do that calculation, or uses a signed JWT to let me have that process cached.
Why shouldn't an automated agent be able to deal with that just as easily, by just feeding that JavaScript to its own interpreter?
My understanding is that it just increases the "expense" of mass crawling just enough to put it out of reach. If it costs fractional pennies per page scrape with just a python or go bot, it costs nickels and dimes to run a headless chromium instance to do the same thing. The purpose is economical - make it too expensive to scrape the "open web". Whether it achieves that goal is another thing.
You have it right. The problem Anubis is intended to solve isn't bots per se, the problem is that bot networks have figured out how to bypass rate limits by sending requests from newly minted, sometimes residential, ip addresses/ranges for each request. Anubis tries to help somewhat by making each (client, address) perform a proof-of-work. For normal users this should be an infrequent inconvenience but for those bot networks they have to do it every time. And if they solve the challenge and keep the cookie then the server "has them" so to speak and can apply ip rate limits normally.
it's a "I don't want my server to be _overrun_ by crawlers" protection which works by
- taking advantage that many crawlers are made very badly/cheaply
- increasing the cost of crawling
thats it, simple but good enough to shake of the dumbest crawlers and to make it worth it for AI agents to e.g. cache site crawling so that they don't craws your site a 1000 times a day but instead just once
The AI crawlers have tens of thousands of IPs and some of them use something akin to a residential botnet.
If they notice that they are getting rate limited or IP blocked, they will use each IP only once. This means that IP based rate limiting simply doesn't work.
The proof of work algorithm in Anubis creates an initial investment that is amortized over multiple requests. If you decide to throw the proof away, you will waste more energy, but if you don't, you can be identified and rate limited.
The automated agent. An never get around this, since running the code is playing by the rules. The goal of the automated agent is to ignore the rules.
I think the only requests it was able to block are plain http requests made over curl or Go's stdlib http client. I see enough of both in httpd logs. Now the cancer has adapted by using a fully featured headless web browser that can complete challenges just like any other client.
As other commenters say, it was completely predictable from the start.
SHA256 is an odd choice given that there are ASICs readily available that can compute hashes crazy fast, made for Bitcoin mining. An ASIC-resistant/memory hard algorithm would be a better choice, possibly one of the Argon2 variants.
Another approach. Require a hash(RESOURCE_ID, ITERATIONS, MEMORY_COST) for each and every resource request. Admittedly that might get a little tricky considering that you don't want to bog down actual users with sluggish page loads. But if carefully tuned to the highest tolerable level it might actually be sufficient. (Maybe.) It's a hard problem....
My dumb idea is to encrypt each HTML element as chain of encryption that requires full load of an HTML/JS element to get another key to load another HTML/JS element and so on. Key retrieval can be throttled and mixed between client and server side and embedded with each requests to prevent browser load everything at once.
This may tread too close to DRM tho due to element protection scheme.
Since there is varying but requester independent input into the hash function, doesnt this mean that the server has to calculate the entire value space too and that these resource hashes can be reused across different requester?
Binding a challange-response to a specific resource doesnt sound like such a bad idea though.
Near as I can guess, the idea is that the code is optimized for what browsers can do and gpus/servers/crawlers/etc can't do as easily (or relatively as easily, just taking up the whole server for a bit might a big cost). Indeed it seems like only a matter of time before something like that would be broken.
Presumably they just finally decided they were willing to spend ($) the CPU time to pass the Anubis check. That was always my understanding of Anubis--of course a bot can pass it, it's just going to cost them a bunch of CPU time (and therefore money) to do it.
If I worked at a billion dollar firm, where doing this was actually a profitable endeavor, I'd reimplement the Anubis algorithm in optimized native code and run that. I wouldn't be surprised if you could lower the cost of generating the proof by a couple of orders of magnitude, enough to make it trivial. If you then batch it, or distribute it across your GPU farm, well now it's practically free.
I've seen a lot of traffic from Huawei bypassing Anubis on some of the things I host as well. The funny thing is, I work for Huawei... Asking around, it seems most of it is coming from Huawei Cloud (like AWS) but their artifactory cache also shows a few other captcha bypassing libraries for Arkose/funcaptcha so they're definitely doing it themselves too.
I wish I had that kind of power. The European labs get funding from HQ on a per project basis and it takes a lot of effort to convince them to do anything. To even contribute code to open source, we need to fill out a bunch of paperwork, much less fund open source projects that work specifically against some other team's objectives. I'd personally just IP block Huawei's entire ASN since much of it is customer controlled and used for scraping. I know a ton of sites are already doing that since I get IP blocked on my work laptop all the time while doing research with their VPN on
Last time I checked, Anubis used SHA256 for PoW. This is very GPU/ASIC friendly, so there's a big disparity between the amount of compute available in a legit browser vs a datacentre-scale scraping operation.
Here's the basic problem: the fully loaded cost of a server CPU core is ~1 cent/hour. The most latency you can afford to inflict on real users is a couple of seconds. That means the cost of passing a challenge the way the users pass it, with a CPU running Javascript, is about 1/1000th of a cent. And then that single proof of work will let them scrape at a minimum hundreds, but more likely thousands, of pages.
So a millionth of a cent per page. How much engineering effort is worth spending on optimizing that? Basically none, certainly not enough to offload to GPUs or ASICs.
No matter where the bar is there will always be scrapers willing to jump over it, but if you can raise the bar while holding the user-facing cost constant, that's a win.
Sorry but it is actually a completely wrong solution. This is not a "make it expensive for spammers" problem. They only need to download the source code once, with a background process that doesn't matter if it takes hours.
Besides the "got the source code for training data" , the other access scenario is just downloading to an end users "agent" Which again, the end user is running something in the background, doesn't care how long it takes, how much it costs, its not a volume or spam type problem
Really feels like this needs some sort of unified possibly legal approach to get these fkers to behave.
Search era clearly proved it is possible to crawl respectfully - the AI crawlers have just decided not to. They need to be disincentivized from doing this
the problem in many cases is that even if such a law is made it likely
- is hard to enforce
- misses bite, i.e. it makes you more money to break it then any penalties
but in general yes, a site which indicates they don't want to be crawled by AI bots but still gets crawled should be handled similar to someone with house ban on a shop forcing them self into the shop
given how severely messed up some millennia cyber security laws are I wonder if crawlers bypassing Anubis could be interpreted as "circumventing digital access controls/protections" or similar, especially given that its done to make copies of copyrighted material ;=)
If you put something in public domain people are going to access it unless you put it behind a paywall but you don't want to do it because that would limit access or people wouldn't pay for it to begin with (ex. your blog nobody wants to pay for)
There's no law against scraping, and we've already past the CFAA argument
Look at it from a lens of harm rather than legality. The hostility comes from people having to pay thousands in bandwidth costs and having services degraded. These AI companies incur huge costs from their wasteful negligence. It's not reasonable.
I really don't get why you'd use an ethnic slur as your HN username. Does it make your secret place all tingly when you imagine those who understand, spot it?
The problem here is that some websites, in this case independent open source code forges, can't be put behind paywalls and cannot maintain availability under the load of scrapers.
Where I live, people often put books in boxes out on the sidewalk as giveaways. They're worth nothing, but I love browsing through them if I have time, and sometimes I find something that does interest me. Books I want to throw away I never throw away, I put them out for others to take. I bet most of them just perish in the rain, but it's fine, it's better than throwing them in the trash right away.
It's something people do for people. It's not "in the public domain" for companies to gobble up with machines.
> There's no law against scraping
There's no law against incurring as heavy and pernicious social and material costs to commercial scrapers as is physically possible within legal bounds, either. So what's the problem?
I feel for Codeberg and people who against AI but also I think Anubis can’t die soon enough. It breaks archiving and is very annoying when JS is disabled or when faced with an aggressive ad-blocker. It breaks web in more ways than one.
How about using on-chain proof-of-work? It flips the script.
If a bot wants access, let it earn it—and let that work be captured, not discarded. Each request becomes compensation to the site itself. The crawler feeds the very system it scrapes. Its computational effort directly funds the site owner's wallet, joining the pool to complete its proof.
The check has to apply to people and bot visitors the same. If you're expecting a blockchain registered spend before the content is visible, basically nobody will visit your website.
I don’t think OP meant you pay directly, just that you volunteer to do some part of the PoW (of some chain designed for this purpose) on behalf of the site, to its credit.
That’s not much of a different ask from Anubis. It just commandeers the compute for some useful purpose.
It would be more like using a javascript webminer to mine for you a bit before you get access. This would be the "proof-of-work" needed to then proceed to scrape.
Which blockchain? An existing mainstream one like Bitcoin?
Because if so, I don't yet see how to "smooth out" the wins. If the crawler manages to solve the very-high-difficulty puzzle for you and get you 1BTC, great, but it will be a long time between wins.
If you're proposing a new (or non-mainstream) blockchain: What makes those coins valuable?
Isn't it the same problem as public mining pools? I remember I ran my little desktop in one for a day or 2 and got paid some micro-coins despite not personally winning a block. I'm not sure how they verify work and prevent cheating, but they appear to do so. I don't know if it scales down to a small enough size to be appropriate for 1 webpage though.
Look into JavaScript web miners for Monero. It exists but just hasn't been implemented into a proof-of-work concept to make a bot/AI scraper "pay" to access your site. Or if it has been used, not at scale.
Crazy thought but what if you made the work required to access the site equal the work required to host site. Host the public part of the database on something like webtorrent. Render website from db locally. You want to ruin expensive queries? Suit yourself. Not easy, but maybe possible?
It is just sad we are in a time where measures like Anubis is necessary. The author's efforts are admirable, so I don't mean this personally: but Anubis is a bad product IMHO.
It doesn't quite do what it is advertised to do, as evidenced by this post; and it degrades user experience for everybody. And it also stops the website from being indexed by search engines (unless specifically configured otherwise). For example, gitlab.freedesktop.org pages have just disappeared from Google.
I'm not anti-the-tech-behind-AI, but this behavior is just awful, and makes the world worse for everyone. I wish AI companies would instead, I don't know, fund common crawl or something so that they can have a single organization and set of bots collecting all the training data they need and then share it, instead of having a bunch of different AI companies doing duplicated work and resulting in a swath of duplicated requests. Also, I don't understand why they have to make so many requests so often. Why wouldn't like one crawl of each site a day, at a reasonable rate, be enough? It's not like up to the minute info is actually important since LLM training cutoffs are always out of date anyway. I don't get it.
Greed. It's never enough money, never enough data, we must have everything all the time and instantly. It's also human nature it seems, looking at how we consume like there's no tomorrow.
This isn't AI. This is corporations doing things because they have a profit motive. The issue here is the non-human corporations and their complete lack of accountability even if someone brings legal charges against them. Their structure is designd to abstract away responsibility and they behave that way.
Yeah, that's why I said I'm not against AI as a technology, but against the behavior of the corporations currently building it. What I'm confused by (not really confused, I understand its just negligence and not giving a fuck, but, frustrated and confused in a sort of helpless sense of not being able to get into the mindset) is just that while there isn't a profit motive against doing this (obviously) there's also not clearly a profit motive to do it, it seems like they're wasting their own resources too on unnecessarily frequent data collection, and also it'd be cheaper to pool data collection efforts.
Yeah, I don't think we can regulate this problem away personally. Because whatever regulations will be made will either be technically impossible and nonsensical products of people who don't understand what they're regulating that will produce worse side effects (@simonw extracted a great quote from recent Doctorow post on this: https://simonwillison.net/2025/Aug/14/cory-doctorow/) or just increase regulatory capture and corporate-state bonds, or even facilitate corp interests, because the big corps are the ones with economic and lobbying power.
> fund common crawl or something so that they can have a single organization and set of bots collecting all the training data they need and then share it
That, or, they could just respect robots.txt and we could put enforcement penalties for not respecting the web service's request to not be crawled. Granted, we probably need a new standard but all these AI companies are just shitting all over the web, being disrespectful of site owners because who's going to stop them? We need laws.
IMO, if digital information is posted publicly online, it's fair game to be crawled unless that crawl is unreasonably expensive or takes it down for others, because these are non rivalrous resources that are literally already public.
> we could put enforcement penalties for not respecting the web service's request to not be crawled... We need laws.
How would that be enforceable? A central government agency watching network traffic? A means of appealing to a bureaucracy like the FCC? Setting it up so you can sue companies that do it? All of those seem like bad options to me.
laws are inherently national, which the internet is not. by all means write a law that crawlers need to obey robots.txt, but how are you going to make russia or china follow that law?
Really looks like the last solution is a legal one, using the DMCA against them using the digital protection or access control circumvention clause or smth.
DMCA only applies to hosted content and we've established that LLM aren't hosting copyrighted content as there is significant transformation which you would otherwise need to prove yourself by training and replicating their entire model.
There is no legal recourse here, if you don't want AI crawlers accessing your content 1) put it behind a paywall 2) remove from public access
I'm not talking about the output of the LLM here. DMCA is an overreaching law. Here i'm talking about its provisions for access controls and "digital locks", i am not a lawyer but i'm fairly sure you could find some way to categorize Anubis/another software as a digital lock and then sue them on that basis.
The tiniest relief is knowing that models will be distilled and "copied" in smaller models of equal capabilities in ~6/12months, since their output can't be copyrighted and will be used to improve others. Kinda ironic
you can't steal something that is in public domain and one which you make readily available by publishing it online because there is no provable cost of damage to you by someone scraping and training their models.
if you really think what you offer has value, put it in behind a paywall and see how many people will consume it then, probably not a lot.
Making my web resources IPv6-only has solved the problem for me. I don’t consider this a solution for ever, but for now it’s apparently way too modern or complicated for the A-so-called-I companies.
In my experience managing a number of IPv6-only sites for clients, they still get crawled and abused, and this goes back years. If anything, it has gotten worse now with all the LLM/AI nonsense.
How long it will be until there will be nothing left to scrape and this activity ends? I mean, at some point payback from it will become less than 0 because they will be consuming more AI generated stuff than real (even adjusted for their ability to detect and filter it), and the content itself will be of very little value (because all high value one will be already scraped). Anyone tried to estimate how long that might take?
The whole reason of anubis is the bot don't make a damn shit about whether whole data is accessible or not, and even crawl dynamic links in robots.txt in high frequency.
Even wikipedia begged for those damn bot about stopping doing this, the data is already accessible in archive here.
So the problem is a bunch of AI companies mining our web content for training data without asking and without regard for hosters' effort/bandwidth and the users' service quality.
And the proposed remedy is to give them human-labeled data directly in the form of captchas, even more severely degrading the user experience and thus website viability?
Why not ask it to directly mine some bitcoin, or do some protein folding? Let's make proof-of-work challenges proof-of-useful-work challenges. The server could even directly serve status 402 with the challenge.
This was beyond predictable. The monetary cost of proof of work is several orders of magnitude too small to deter scraping (let alone higher yield abuse), and passing the challenges requires no technical finesse basically by construction.
We need to revive 402 Payment Required, clearly. If we lived in a world where we could easily set up a small trusted online balance for microtransactions that's interoperable with everyone, and where giving others a literal penny for their thoughts could allow for running up a significant bill for abusers, I'd gladly play along.
Me too. I wouldn't mind Project Xanadu style micro payments for blogs, and it'd both fix the AI scraper issue and the ads issue, and help people fund hosting costs sustainably. I think the issue is taxes and transaction fees would push the prices too high, and it'd price out people with very low income possibly. It'd also create really perverse incentives for even more tight copyright control, since your content appearing even in part on anyone else's website is then directly losing you money, so it'd destroy the public Commons even more, which would be bad. But maybe not, who knows.
I would definitely use this to charge US$100,000 per request from any AI company to crawl my site. I would exempt 'public good' crawlers like The Internet Archive though.
If AI companies valued at billions of dollars want to slurp up my contribution to the human condition, that's my price - subject to price rises only.
For someone doing spamming that low level would work well. As their cost is determinatively low to make it work. For someone doing scraping to get data and feeding it to an AI not so much. The AI groups usually have some pretty heavy hitting hardware sitting behind it. They could even break off some hardware that is to be retired and have it munch away on it. To make it non cost effective the calculations would need to be much bigger.
anubis and others allow some user agents to pass without proof of work. bad bots (and user) just use an extension that detect anubis and change the user agent instead.
it's well intentioned but just waste electricity from good people in the end.
anubis does nothing to impact bad crawlers, well only the laziest ones. but for those generating fake infinite content on the fly is much more efficient.
I'm calling it now, this is the beginning of all of the remaining non-commerical properties on the web either going away, or getting hidden inside of some trusted overlay network. Unless the "AI" race slows down or changes or some other act of god happens, the incentives are aligned that I foresee wide swaths of the net getting flogged to death.
Also increasing balkanization of the internet. I now routinely run into sites that geoblock my whole country, this wasn't something I would see more than once or twice a year, and usually only with sites like Walmart that don't care about clients from outside the US.
Now it's 2-5 sites per day, including web forums and such.
I self-host a few servers and have not seen significant traffic increases from crawlers, so I can't agree with that without seeing some evidence of this issue's scale and scope. As far as I know it mostly affects commercial content aggregators.
Hasn’t that been the case for a while? I’d imagine the combined traffic to all sites on the web combined doesn’t match a single hour of the traffic to the top 5 social media sites. The web is pretty much dead for a while now, many companies don’t even bother maintaining websites anymore
> [...] Now it’s LLMs. If you think these crawlers respect robots.txt then you are several assumptions of good faith removed from reality. These bots crawl everything they can find, robots.txt be damned, including expensive endpoints like git blame, every page of every git log, and every commit in every repo, and they do so using random User-Agents that overlap with end-users and come from tens of thousands of IP addresses – mostly residential, in unrelated subnets, each one making no more than one HTTP request over any time period we tried to measure – actively and maliciously adapting and blending in with end-user traffic and avoiding attempts to characterize their behavior or block their traffic.
The linux kernel has also been dealing with it AFAIK. Apparently it's not so easy to deal with, because these ai scrapers pull a lot of tricks to anonymize themselves.
One of my semi-personal websites gets crawled by AI crawlers a ton now. I use Bunny.net for a cdn. $20 used to last me for months of traffic, now it only lasts a week or two at most. It's enough that I'm going to go back to not using a cdn and just let the site suffer some slowness every once in a while.
I could see it being the end of commercial and institutional web applications which cannot handle traffic. But actual websites which are html and files in folders served by webservers don't have problems with this.
I think there was some sort of fake webserver that did something like this already. Basically just linked endlessly to more llm-generated pages of nonsense.
There are lots more as well; those are just a few of the ones that recently made the rounds.
I suspect that combining approaches will be a tractable way to waste time:
- Anubis-esque systems to defeat or delay easily-deterred or cut-rate crawlers,
- CloudFlare or similar for more invasive-to-real-humans crawler deterrence (perhaps only served to a fraction of traffic or traffic that crosses a suspicion threshold?),
- Junk content rings like Nepenthes as honeypots or "A/B tests" for whether a particular traffic type is an AI or not (if it keeps following nonsense-content links endlessly, it's not a human; if it gives up pretty quickly, it might be--this costs/pisses off users but can be used as a test to better train traffic-analysis rules that trigger the other approaches on this list in response to detected likely-crawler traffic).
- Model poisoners out of sheer pettiness, if it brings you joy.
I also wonder if serving taboo traffic (e.g. legal but beyond-the-pale for most commercial applications porn/erotica) would deter some AI crawlers. There might be front-side content filters that either blacklist or de-prioritize sites whose main content appears (to the crawler) to be at some intersection of inappropriate, prohibited, and not widely-enough related to model output as to be in demand.
This is sad, but predictable. At the end of the day if I can follow a link to an Anubis protected site and view it on my phone, the crawlers will be able to as well.
I see a lot more private networks in our future, unfortunately.
The private network is only as good as the weakest link which has to offer a reason for people to go through the trouble of accessing either by paying money or other means (acquiring special equipment).
And by putting a wall up you end up losing a large portion of the market to those that will now simply arbitrage and fill the space you leave behind.
There is simply no way to stop crawlers/scrapers, period, unless you put a meter on it or go offline.
The people that are already using Anubis don’t care about “losing a large portion of the market” we just want to work on our FOSS projects without paying unnecessary bills due to out of control crawling.
Duh. The author of Anubis really should advertise it as a DDoS guard, not an AI guard. Otherwise, xe is just misleading people, while being unnecessarily discriminatory against robots (robotkin) and cyborgs (those using AI agents as an extension of their selves).
It's not really a DDoS guard though. If someone wants to ddos a server, anubis isn't going to be able to stop the traffic before it gets to the server.
It does help from accidental ddos or just rude scrapers that assume everyone has unlimited bandwidth and money.
We’ve evolved. The only thing that pays attention to the OSI model is DDoS which can hit you on every layer. And against the application layer attacks Anubis and friends are effective.
I just found out about this when it came to the front page of Hacker News. I really wish I was given advanced notice. I haven't been able to put as much energy into Anubis as I've wanted because I've been incredibly overwhelmed by life and need to be able to afford to make this my full time job. Support contracts are being roadblocked, and I just wish I had the time and energy to focus on this without having to worry about being the single income for the household.
I don't think Xena is going to promote themselves but I looked at their GitHub sponsor's page. They have a monthly goal of $5k but haven't even hit $1.5k. Combined with their Pateron they get a whopping $2k/mo.
There's a lot of love for Anubis alone, but this doesn't translate to a third of what a junior dev makes. How do we expect to turn these highly beneficial open source products when people cannot pass off a small portion of their savings to those that make the tools.
I hope you figure it out Xena. Even though I'm only on the user side of things I've liked Anubis more than other solutions like cloudflare.
https://github.com/sponsors/Xe#sponsors
https://www.patreon.com/cadey
.
3 replies →
That sucks. Keep fighting the good fight, and I wish you all the best. We need people working on this problem (unfortunately).
Thanks! I just wish I could afford to work on this full time, or at least even part time. It would help me a lot and prevent me from having to work what is effectively two full time jobs. Rent and food keep getting more expensive in Canada.
Good luck, I'm sorry for all of this speculation and people attacking your solution instead of suggesting concrete improvements to help fight the problem.
Thanks. It means a lot. Today has not been a good day for me. It will be fixed. Things will get better, but this has to rank up there in terms of the worst ways to find out about security issues. It sucks lol.
6 replies →
I actually don't understand who Anubis is supposed to "make sure you're not a bot". It seems to be more of a rate limiter than anything else. It self-describes:
> Anubis sits in the background and weighs the risk of incoming requests. If it asks a client to complete a challenge, no user interaction is required.
> Anubis uses a proof-of-work challenge to ensure that clients are using a modern browser and are able to calculate SHA-256 checksums. Anubis has a customizable difficulty for this proof-of-work challenge, but defaults to 5 leading zeroes.
When I go to Codeberg or any other site using it, I'm never asked to perform any kind of in-browser task. It just has my browser run some JavaScript to do that calculation, or uses a signed JWT to let me have that process cached.
Why shouldn't an automated agent be able to deal with that just as easily, by just feeding that JavaScript to its own interpreter?
My understanding is that it just increases the "expense" of mass crawling just enough to put it out of reach. If it costs fractional pennies per page scrape with just a python or go bot, it costs nickels and dimes to run a headless chromium instance to do the same thing. The purpose is economical - make it too expensive to scrape the "open web". Whether it achieves that goal is another thing.
what do AI companies have more than everyone else? compute
anubis directly incentivises the adversary, at expense of everyone else
it's what you would deploy if you want to exclude everyone else
(conspiracy theorists note that the author worked for an AI firm)
7 replies →
You have it right. The problem Anubis is intended to solve isn't bots per se, the problem is that bot networks have figured out how to bypass rate limits by sending requests from newly minted, sometimes residential, ip addresses/ranges for each request. Anubis tries to help somewhat by making each (client, address) perform a proof-of-work. For normal users this should be an infrequent inconvenience but for those bot networks they have to do it every time. And if they solve the challenge and keep the cookie then the server "has them" so to speak and can apply ip rate limits normally.
it's indeed not a "bot/crawler protection"
it's a "I don't want my server to be _overrun_ by crawlers" protection which works by
- taking advantage that many crawlers are made very badly/cheaply
- increasing the cost of crawling
thats it, simple but good enough to shake of the dumbest crawlers and to make it worth it for AI agents to e.g. cache site crawling so that they don't craws your site a 1000 times a day but instead just once
The AI crawlers have tens of thousands of IPs and some of them use something akin to a residential botnet.
If they notice that they are getting rate limited or IP blocked, they will use each IP only once. This means that IP based rate limiting simply doesn't work.
The proof of work algorithm in Anubis creates an initial investment that is amortized over multiple requests. If you decide to throw the proof away, you will waste more energy, but if you don't, you can be identified and rate limited.
The automated agent. An never get around this, since running the code is playing by the rules. The goal of the automated agent is to ignore the rules.
I think the only requests it was able to block are plain http requests made over curl or Go's stdlib http client. I see enough of both in httpd logs. Now the cancer has adapted by using a fully featured headless web browser that can complete challenges just like any other client.
As other commenters say, it was completely predictable from the start.
SHA256 is an odd choice given that there are ASICs readily available that can compute hashes crazy fast, made for Bitcoin mining. An ASIC-resistant/memory hard algorithm would be a better choice, possibly one of the Argon2 variants.
Another approach. Require a hash(RESOURCE_ID, ITERATIONS, MEMORY_COST) for each and every resource request. Admittedly that might get a little tricky considering that you don't want to bog down actual users with sluggish page loads. But if carefully tuned to the highest tolerable level it might actually be sufficient. (Maybe.) It's a hard problem....
My dumb idea is to encrypt each HTML element as chain of encryption that requires full load of an HTML/JS element to get another key to load another HTML/JS element and so on. Key retrieval can be throttled and mixed between client and server side and embedded with each requests to prevent browser load everything at once.
This may tread too close to DRM tho due to element protection scheme.
Since there is varying but requester independent input into the hash function, doesnt this mean that the server has to calculate the entire value space too and that these resource hashes can be reused across different requester?
Binding a challange-response to a specific resource doesnt sound like such a bad idea though.
1 reply →
Near as I can guess, the idea is that the code is optimized for what browsers can do and gpus/servers/crawlers/etc can't do as easily (or relatively as easily, just taking up the whole server for a bit might a big cost). Indeed it seems like only a matter of time before something like that would be broken.
Presumably they just finally decided they were willing to spend ($) the CPU time to pass the Anubis check. That was always my understanding of Anubis--of course a bot can pass it, it's just going to cost them a bunch of CPU time (and therefore money) to do it.
If I worked at a billion dollar firm, where doing this was actually a profitable endeavor, I'd reimplement the Anubis algorithm in optimized native code and run that. I wouldn't be surprised if you could lower the cost of generating the proof by a couple of orders of magnitude, enough to make it trivial. If you then batch it, or distribute it across your GPU farm, well now it's practically free.
I think so too. Maybe the compute cost needs to be upped some more. I am OK with waiting a bit longer when I access the site.
I've seen a lot of traffic from Huawei bypassing Anubis on some of the things I host as well. The funny thing is, I work for Huawei... Asking around, it seems most of it is coming from Huawei Cloud (like AWS) but their artifactory cache also shows a few other captcha bypassing libraries for Arkose/funcaptcha so they're definitely doing it themselves too.
Anonymous account for obvious reasons.
Please have someone in the common sense department email me@xeiaso.net. Funding of the Anubis project would go a long way towards mending bridges.
I wish I had that kind of power. The European labs get funding from HQ on a per project basis and it takes a lot of effort to convince them to do anything. To even contribute code to open source, we need to fill out a bunch of paperwork, much less fund open source projects that work specifically against some other team's objectives. I'd personally just IP block Huawei's entire ASN since much of it is customer controlled and used for scraping. I know a ton of sites are already doing that since I get IP blocked on my work laptop all the time while doing research with their VPN on
Who exactly is Huawei corporate interested in mending bridges with? Seems like that tie is long severed
Last time I checked, Anubis used SHA256 for PoW. This is very GPU/ASIC friendly, so there's a big disparity between the amount of compute available in a legit browser vs a datacentre-scale scraping operation.
A more memory-hard "mining" algorithm could help.
A different algorithm would not help.
Here's the basic problem: the fully loaded cost of a server CPU core is ~1 cent/hour. The most latency you can afford to inflict on real users is a couple of seconds. That means the cost of passing a challenge the way the users pass it, with a CPU running Javascript, is about 1/1000th of a cent. And then that single proof of work will let them scrape at a minimum hundreds, but more likely thousands, of pages.
So a millionth of a cent per page. How much engineering effort is worth spending on optimizing that? Basically none, certainly not enough to offload to GPUs or ASICs.
No matter where the bar is there will always be scrapers willing to jump over it, but if you can raise the bar while holding the user-facing cost constant, that's a win.
3 replies →
Sorry but it is actually a completely wrong solution. This is not a "make it expensive for spammers" problem. They only need to download the source code once, with a background process that doesn't matter if it takes hours.
Besides the "got the source code for training data" , the other access scenario is just downloading to an end users "agent" Which again, the end user is running something in the background, doesn't care how long it takes, how much it costs, its not a volume or spam type problem
Really feels like this needs some sort of unified possibly legal approach to get these fkers to behave.
Search era clearly proved it is possible to crawl respectfully - the AI crawlers have just decided not to. They need to be disincentivized from doing this
the problem in many cases is that even if such a law is made it likely
- is hard to enforce
- misses bite, i.e. it makes you more money to break it then any penalties
but in general yes, a site which indicates they don't want to be crawled by AI bots but still gets crawled should be handled similar to someone with house ban on a shop forcing them self into the shop
given how severely messed up some millennia cyber security laws are I wonder if crawlers bypassing Anubis could be interpreted as "circumventing digital access controls/protections" or similar, especially given that its done to make copies of copyrighted material ;=)
I really don't get this type of hostility
If you put something in public domain people are going to access it unless you put it behind a paywall but you don't want to do it because that would limit access or people wouldn't pay for it to begin with (ex. your blog nobody wants to pay for)
There's no law against scraping, and we've already past the CFAA argument
Look at it from a lens of harm rather than legality. The hostility comes from people having to pay thousands in bandwidth costs and having services degraded. These AI companies incur huge costs from their wasteful negligence. It's not reasonable.
It’s not quite as simple as “putting something in public domain”. The problem is the server costs to keep that thing in the public domain.
I really don't get why you'd use an ethnic slur as your HN username. Does it make your secret place all tingly when you imagine those who understand, spot it?
The problem here is that some websites, in this case independent open source code forges, can't be put behind paywalls and cannot maintain availability under the load of scrapers.
Where I live, people often put books in boxes out on the sidewalk as giveaways. They're worth nothing, but I love browsing through them if I have time, and sometimes I find something that does interest me. Books I want to throw away I never throw away, I put them out for others to take. I bet most of them just perish in the rain, but it's fine, it's better than throwing them in the trash right away.
It's something people do for people. It's not "in the public domain" for companies to gobble up with machines.
> There's no law against scraping
There's no law against incurring as heavy and pernicious social and material costs to commercial scrapers as is physically possible within legal bounds, either. So what's the problem?
I feel for Codeberg and people who against AI but also I think Anubis can’t die soon enough. It breaks archiving and is very annoying when JS is disabled or when faced with an aggressive ad-blocker. It breaks web in more ways than one.
How about using on-chain proof-of-work? It flips the script.
If a bot wants access, let it earn it—and let that work be captured, not discarded. Each request becomes compensation to the site itself. The crawler feeds the very system it scrapes. Its computational effort directly funds the site owner's wallet, joining the pool to complete its proof.
The cost becomes the contract.
The check has to apply to people and bot visitors the same. If you're expecting a blockchain registered spend before the content is visible, basically nobody will visit your website.
I don’t think OP meant you pay directly, just that you volunteer to do some part of the PoW (of some chain designed for this purpose) on behalf of the site, to its credit.
That’s not much of a different ask from Anubis. It just commandeers the compute for some useful purpose.
5 replies →
It would be more like using a javascript webminer to mine for you a bit before you get access. This would be the "proof-of-work" needed to then proceed to scrape.
The last thing I want on the web is the normalization of cryptominers.
Then again, if you're doing a useless computation one way or another, it might as well benefit the website whose content you're requesting
But in principle I very much agree
Which blockchain? An existing mainstream one like Bitcoin?
Because if so, I don't yet see how to "smooth out" the wins. If the crawler manages to solve the very-high-difficulty puzzle for you and get you 1BTC, great, but it will be a long time between wins.
If you're proposing a new (or non-mainstream) blockchain: What makes those coins valuable?
Isn't it the same problem as public mining pools? I remember I ran my little desktop in one for a day or 2 and got paid some micro-coins despite not personally winning a block. I'm not sure how they verify work and prevent cheating, but they appear to do so. I don't know if it scales down to a small enough size to be appropriate for 1 webpage though.
1 reply →
Look into JavaScript web miners for Monero. It exists but just hasn't been implemented into a proof-of-work concept to make a bot/AI scraper "pay" to access your site. Or if it has been used, not at scale.
Crazy thought but what if you made the work required to access the site equal the work required to host site. Host the public part of the database on something like webtorrent. Render website from db locally. You want to ruin expensive queries? Suit yourself. Not easy, but maybe possible?
It is just sad we are in a time where measures like Anubis is necessary. The author's efforts are admirable, so I don't mean this personally: but Anubis is a bad product IMHO.
It doesn't quite do what it is advertised to do, as evidenced by this post; and it degrades user experience for everybody. And it also stops the website from being indexed by search engines (unless specifically configured otherwise). For example, gitlab.freedesktop.org pages have just disappeared from Google.
We need to find a better way.
I'm not anti-the-tech-behind-AI, but this behavior is just awful, and makes the world worse for everyone. I wish AI companies would instead, I don't know, fund common crawl or something so that they can have a single organization and set of bots collecting all the training data they need and then share it, instead of having a bunch of different AI companies doing duplicated work and resulting in a swath of duplicated requests. Also, I don't understand why they have to make so many requests so often. Why wouldn't like one crawl of each site a day, at a reasonable rate, be enough? It's not like up to the minute info is actually important since LLM training cutoffs are always out of date anyway. I don't get it.
Greed. It's never enough money, never enough data, we must have everything all the time and instantly. It's also human nature it seems, looking at how we consume like there's no tomorrow.
It doesn't even make sense to crawl this way. It's just destructive for almost no beinifit.
2 replies →
Which is why internalizing externalities is so important, but that's also extremely hard to do right (leads to a lot of "nerd harder" problems).
This isn't AI. This is corporations doing things because they have a profit motive. The issue here is the non-human corporations and their complete lack of accountability even if someone brings legal charges against them. Their structure is designd to abstract away responsibility and they behave that way.
Same old problem. Corps are gonna corp.
Yeah, that's why I said I'm not against AI as a technology, but against the behavior of the corporations currently building it. What I'm confused by (not really confused, I understand its just negligence and not giving a fuck, but, frustrated and confused in a sort of helpless sense of not being able to get into the mindset) is just that while there isn't a profit motive against doing this (obviously) there's also not clearly a profit motive to do it, it seems like they're wasting their own resources too on unnecessarily frequent data collection, and also it'd be cheaper to pool data collection efforts.
The time to regulate tech was like 15 years ago, and we didn't. Why would any tech company expect to have to start following "rules" now?
Yeah, I don't think we can regulate this problem away personally. Because whatever regulations will be made will either be technically impossible and nonsensical products of people who don't understand what they're regulating that will produce worse side effects (@simonw extracted a great quote from recent Doctorow post on this: https://simonwillison.net/2025/Aug/14/cory-doctorow/) or just increase regulatory capture and corporate-state bonds, or even facilitate corp interests, because the big corps are the ones with economic and lobbying power.
> fund common crawl or something so that they can have a single organization and set of bots collecting all the training data they need and then share it
That, or, they could just respect robots.txt and we could put enforcement penalties for not respecting the web service's request to not be crawled. Granted, we probably need a new standard but all these AI companies are just shitting all over the web, being disrespectful of site owners because who's going to stop them? We need laws.
> That, or, they could just respect robots.txt
IMO, if digital information is posted publicly online, it's fair game to be crawled unless that crawl is unreasonably expensive or takes it down for others, because these are non rivalrous resources that are literally already public.
> we could put enforcement penalties for not respecting the web service's request to not be crawled... We need laws.
How would that be enforceable? A central government agency watching network traffic? A means of appealing to a bureaucracy like the FCC? Setting it up so you can sue companies that do it? All of those seem like bad options to me.
4 replies →
laws are inherently national, which the internet is not. by all means write a law that crawlers need to obey robots.txt, but how are you going to make russia or china follow that law?
if those companies cared about acting in good faith, they wouldnt be in AI
Really looks like the last solution is a legal one, using the DMCA against them using the digital protection or access control circumvention clause or smth.
DMCA only applies to hosted content and we've established that LLM aren't hosting copyrighted content as there is significant transformation which you would otherwise need to prove yourself by training and replicating their entire model.
There is no legal recourse here, if you don't want AI crawlers accessing your content 1) put it behind a paywall 2) remove from public access
I'm not talking about the output of the LLM here. DMCA is an overreaching law. Here i'm talking about its provisions for access controls and "digital locks", i am not a lawyer but i'm fairly sure you could find some way to categorize Anubis/another software as a digital lock and then sue them on that basis.
thanks for making everything that much shittier just so you can steal everyone's data and present it as your own, AI companies!
The tiniest relief is knowing that models will be distilled and "copied" in smaller models of equal capabilities in ~6/12months, since their output can't be copyrighted and will be used to improve others. Kinda ironic
Dear god that invokes the image of some un-human centipede...
you can't steal something that is in public domain and one which you make readily available by publishing it online because there is no provable cost of damage to you by someone scraping and training their models.
if you really think what you offer has value, put it in behind a paywall and see how many people will consume it then, probably not a lot.
Making my web resources IPv6-only has solved the problem for me. I don’t consider this a solution for ever, but for now it’s apparently way too modern or complicated for the A-so-called-I companies.
In my experience managing a number of IPv6-only sites for clients, they still get crawled and abused, and this goes back years. If anything, it has gotten worse now with all the LLM/AI nonsense.
How long it will be until there will be nothing left to scrape and this activity ends? I mean, at some point payback from it will become less than 0 because they will be consuming more AI generated stuff than real (even adjusted for their ability to detect and filter it), and the content itself will be of very little value (because all high value one will be already scraped). Anyone tried to estimate how long that might take?
What do these crawlers gather? Just make this data accessible via API calls or direct database download, like Wikipedia did (https://en.wikipedia.org/wiki/Wikipedia:Database_download).
The whole reason of anubis is the bot don't make a damn shit about whether whole data is accessible or not, and even crawl dynamic links in robots.txt in high frequency.
Even wikipedia begged for those damn bot about stopping doing this, the data is already accessible in archive here.
Are AI crawlers equipped to get past reCAPTCHA or hCAPTCHA? This seems like exactly the thing these services were meant to stop.
So the problem is a bunch of AI companies mining our web content for training data without asking and without regard for hosters' effort/bandwidth and the users' service quality.
And the proposed remedy is to give them human-labeled data directly in the form of captchas, even more severely degrading the user experience and thus website viability?
Color me unconvinced.
Why not ask it to directly mine some bitcoin, or do some protein folding? Let's make proof-of-work challenges proof-of-useful-work challenges. The server could even directly serve status 402 with the challenge.
https://news.ycombinator.com/item?id=44880393
Note that the work needs to produce a result that's quickly verifiable by the server.
Do you want people to mine bitcoin or do protein folding to read your blog or access your web application?
More importantly do you want to now compete with those that do not bottleneck and lose your traffic ?
This is the paradox, the length you go to protect your content only increases costs for everybody else who isn't an AI crawler.
People, no! Robots which can't pass for people, yes.
This was beyond predictable. The monetary cost of proof of work is several orders of magnitude too small to deter scraping (let alone higher yield abuse), and passing the challenges requires no technical finesse basically by construction.
We need to revive 402 Payment Required, clearly. If we lived in a world where we could easily set up a small trusted online balance for microtransactions that's interoperable with everyone, and where giving others a literal penny for their thoughts could allow for running up a significant bill for abusers, I'd gladly play along.
Me too. I wouldn't mind Project Xanadu style micro payments for blogs, and it'd both fix the AI scraper issue and the ads issue, and help people fund hosting costs sustainably. I think the issue is taxes and transaction fees would push the prices too high, and it'd price out people with very low income possibly. It'd also create really perverse incentives for even more tight copyright control, since your content appearing even in part on anyone else's website is then directly losing you money, so it'd destroy the public Commons even more, which would be bad. But maybe not, who knows.
Cloudflare are working on this at the moment.
https://blog.cloudflare.com/introducing-pay-per-crawl/
There's also an open specification called x402:
https://www.x402.org/x402-whitepaper.pdf
I would definitely use this to charge US$100,000 per request from any AI company to crawl my site. I would exempt 'public good' crawlers like The Internet Archive though.
If AI companies valued at billions of dollars want to slurp up my contribution to the human condition, that's my price - subject to price rises only.
Pay to visit would be great, and would force these AI companies to actually pay for their data.
For someone doing spamming that low level would work well. As their cost is determinatively low to make it work. For someone doing scraping to get data and feeding it to an AI not so much. The AI groups usually have some pretty heavy hitting hardware sitting behind it. They could even break off some hardware that is to be retired and have it munch away on it. To make it non cost effective the calculations would need to be much bigger.
anubis and others allow some user agents to pass without proof of work. bad bots (and user) just use an extension that detect anubis and change the user agent instead.
it's well intentioned but just waste electricity from good people in the end.
anubis does nothing to impact bad crawlers, well only the laziest ones. but for those generating fake infinite content on the fly is much more efficient.
It's PoW, AI crawlers didn't learn shit, their admins just increased their CPU/GPU/ASIC budget
I'm calling it now, this is the beginning of all of the remaining non-commerical properties on the web either going away, or getting hidden inside of some trusted overlay network. Unless the "AI" race slows down or changes or some other act of god happens, the incentives are aligned that I foresee wide swaths of the net getting flogged to death.
Also increasing balkanization of the internet. I now routinely run into sites that geoblock my whole country, this wasn't something I would see more than once or twice a year, and usually only with sites like Walmart that don't care about clients from outside the US.
Now it's 2-5 sites per day, including web forums and such.
If you live in Europe it probably has more to do with over-regulation than anything AI-related.
2 replies →
I self-host a few servers and have not seen significant traffic increases from crawlers, so I can't agree with that without seeing some evidence of this issue's scale and scope. As far as I know it mostly affects commercial content aggregators.
It affects many open source projects as well, they just scrape everything repeatedly without abandon.
First from known networks, then from residential IPs. First with dumb http clients, now with full blown headless chrome browsers.
1 reply →
Hasn’t that been the case for a while? I’d imagine the combined traffic to all sites on the web combined doesn’t match a single hour of the traffic to the top 5 social media sites. The web is pretty much dead for a while now, many companies don’t even bother maintaining websites anymore
I think the answer for the non-commercial web is to stop worrying.
I understand why certain business models have a problem with AI crawlers, but I fail to see why sites like Codeberg have an issue.
If the problem is cost for the traffic then this is nothing new and I thought we have learned how to handle that by now.
The issue is the insane amount of traffic from crawlers that DDOS websites.
For example: https://drewdevault.com/2025/03/17/2025-03-17-Stop-externali...
> [...] Now it’s LLMs. If you think these crawlers respect robots.txt then you are several assumptions of good faith removed from reality. These bots crawl everything they can find, robots.txt be damned, including expensive endpoints like git blame, every page of every git log, and every commit in every repo, and they do so using random User-Agents that overlap with end-users and come from tens of thousands of IP addresses – mostly residential, in unrelated subnets, each one making no more than one HTTP request over any time period we tried to measure – actively and maliciously adapting and blending in with end-user traffic and avoiding attempts to characterize their behavior or block their traffic.
The linux kernel has also been dealing with it AFAIK. Apparently it's not so easy to deal with, because these ai scrapers pull a lot of tricks to anonymize themselves.
2 replies →
About 3 hours ago the codeberg website was really slow.
Services like codeberg that are run on donations can be easily DOS'ed by AI crawlers
One of my semi-personal websites gets crawled by AI crawlers a ton now. I use Bunny.net for a cdn. $20 used to last me for months of traffic, now it only lasts a week or two at most. It's enough that I'm going to go back to not using a cdn and just let the site suffer some slowness every once in a while.
I could see it being the end of commercial and institutional web applications which cannot handle traffic. But actual websites which are html and files in folders served by webservers don't have problems with this.
Could it be a 'correct' continuation of Darwin's survival of the fittest?
Fight fire with fire by serving these guys LLM output of made-up news. Wish them good luck noticing that in their dataset.
I think there was some sort of fake webserver that did something like this already. Basically just linked endlessly to more llm-generated pages of nonsense.
There are several!
Some focus on generating content that can be served to waste crawler time: crates.io/crates/iocaine/2.1.0
Some focus on generating linked pages: https://hackaday.com/2025/01/23/trap-naughty-web-crawlers-in...
Some of them play the long game and try to poison models' data: https://codeberg.org/konterfai/konterfai
There are lots more as well; those are just a few of the ones that recently made the rounds.
I suspect that combining approaches will be a tractable way to waste time:
- Anubis-esque systems to defeat or delay easily-deterred or cut-rate crawlers,
- CloudFlare or similar for more invasive-to-real-humans crawler deterrence (perhaps only served to a fraction of traffic or traffic that crosses a suspicion threshold?),
- Junk content rings like Nepenthes as honeypots or "A/B tests" for whether a particular traffic type is an AI or not (if it keeps following nonsense-content links endlessly, it's not a human; if it gives up pretty quickly, it might be--this costs/pisses off users but can be used as a test to better train traffic-analysis rules that trigger the other approaches on this list in response to detected likely-crawler traffic).
- Model poisoners out of sheer pettiness, if it brings you joy.
I also wonder if serving taboo traffic (e.g. legal but beyond-the-pale for most commercial applications porn/erotica) would deter some AI crawlers. There might be front-side content filters that either blacklist or de-prioritize sites whose main content appears (to the crawler) to be at some intersection of inappropriate, prohibited, and not widely-enough related to model output as to be in demand.
This is sad, but predictable. At the end of the day if I can follow a link to an Anubis protected site and view it on my phone, the crawlers will be able to as well.
I see a lot more private networks in our future, unfortunately.
The private network is only as good as the weakest link which has to offer a reason for people to go through the trouble of accessing either by paying money or other means (acquiring special equipment).
And by putting a wall up you end up losing a large portion of the market to those that will now simply arbitrage and fill the space you leave behind.
There is simply no way to stop crawlers/scrapers, period, unless you put a meter on it or go offline.
The people that are already using Anubis don’t care about “losing a large portion of the market” we just want to work on our FOSS projects without paying unnecessary bills due to out of control crawling.
They failed to properly block/throttle the IP subnet as per their admission, and are now blaming others for their failure.
Duh. The author of Anubis really should advertise it as a DDoS guard, not an AI guard. Otherwise, xe is just misleading people, while being unnecessarily discriminatory against robots (robotkin) and cyborgs (those using AI agents as an extension of their selves).
It's not really a DDoS guard though. If someone wants to ddos a server, anubis isn't going to be able to stop the traffic before it gets to the server.
It does help from accidental ddos or just rude scrapers that assume everyone has unlimited bandwidth and money.
We’ve evolved. The only thing that pays attention to the OSI model is DDoS which can hit you on every layer. And against the application layer attacks Anubis and friends are effective.