Comment by AussieWog93
1 day ago
I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots?
WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that.
Are you using lambdas or something like that to serve a blog?
Is it basically just OCD? Vulnerability protection? Old habits from a time where crawling really did impact service?
In my case it's my Forgejo instance. My blog is static files with a limited amount of pages to crawl and indeed it's no issue at all. Forgejo is a dynamic service with an infinite amount of pages to discover and it runs git in the background to generate (some of) the pages. Tons of bots can easily overwhelm my small server. I could hide the repositories, but they're open on purpose (it's open source after all).
I have it guarded by a simple cookie check now, which seems to work well enough. Only a small subset of bots pass it. It likely kills search engine discoverability, but that's the tradeoff I have to make.
That seems fine. You are using a proportionate response to an actual problem.
I have a personal site on a shared host that was recently suspended for excessive CPU use caused by incessant AI bot crawling. I don't have a problem with crawling per se, there's just so many and they're so bad at it.
At home, I can't afford a 40gbit connection and a server to match... getting DDoSed by a handful of malicious google cloud VPSes running nmap and various web vulnerability scans can easily impact performance on low end hardware.
Yes, in the grand scheme of things, waiting a few additional seconds to check my local imap server (because my DMZ gets bogged down) isn't a huge deal, but that doesn't mean I have to like it, or allow it to continue.
Are you getting 40gbits of bot traffic?
For me it's just a fun exercise to see what generalizations I can make about bots that botters can not really avoid or code around such as javascript. Yeah this blog is all precompressed static content in a ram disk. I'm sure it could handle a few hundred thousand requests per second. This is to show methods people could use on their forums, chan sites, chat servers, etc... Every option is optional and tunable. Any option people consider should be tested on a test server. People could just point and laugh if they so desired and that works for me too.
As of now roughly 4661 real humans can reach it. 9 people can not reach it. A few of those 9 I expect to be blocked being on VPN's that egress data-centers.
I could improve this by moving most of the blocking to a demo site so the article is easier to access. I will do that. I will also update some of the nginx ACL's to what I am currently using that explain to the botters what is out of place.
As of now 6622 real humans made it through and a small number of what look to be bots. Curious if any of these people will try their own experiments on this topic.
Bender, even got yourself a great username. One of the most responsive authors in memory.
1 reply →
Not pointing and laughing, just wondering why (since I see so many posts about this). Sounds like it's just for the thrill of the game, which is completely valid.
Thrill of the game is a great way to put it.
1 reply →
i guess the thing i'm most confused about is what is the higher level goal here. 1 in 8 humans on the planet are experiencing the "web" through chatgpt alone. many have migrated to purely agentic workflows.
is the goal for your content to just be invisible to this growing population? is the expectation that all of this is just a fad, which will fade away? what is the end game to the tactics you have outlined? what is the strategy?
I like those ideas. If people are truly operating purely through LLM's then I am fine with being partitioned from them. The LLM operators will provide their reality and their truth. This is all for low trust internet-wide access.
For smaller higher trust communities I don't do any of this. Rather we use basic authentication to keep bots and strangers off the services all together and then regular user accounts on forums and such. That is where strategy comes into play and has been working well for some time.
2 replies →
Do you have a source for those numbers?
5 replies →
I wonder what the click through rate is for the source links. Based on what I've been hearing, it's very low. And that's if the particular LLM/interface/answer even bothers to add them.
If the bots are eating your site traffic and users, what's the point of allowing them in?
Bots routinely overwhelm my public-facing websites. The last few times I've turned on "attack mode" in Cloudflare (which I do when bot swarms get particularly nasty), traffic dropped by at least 20k requests/minute (which is huge, considering my human traffic is around 3/4 of that).
It's mainly that dealing with them is a hassle, time that could be better spent doing other more useful things.
Over the weekend I got round to turning off some old viewvc (CVS and subversion) and hgweb (Mercurial) web interfaces. They'd been running fine for a decade or two with basic security updates etc, mostly only occasionally visited for real purposes, but that's fine!
But getting 2.7 million requests/day from residential proxy IPs was a bit taxing for the old uwsgi/cgi programs (that's average 30/s), making other sites on the server suffer. It was also getting close to 1TB/month which is the VPS quota.
So I ended up turning them off, one more cut towards the centralised internet. Sure I could put a cache in front, but there are probably millions of combinations of dynamic urls in the VCS tree so caching mightn't help much anyway, not worth the effort. I'm sure the web server could be tuned better too, but again it's a waste of time.
30/s is preposterous. Are these loser companies who don’t want to cache on their end or are they getting unique info per request or could it really be that many different origins or…?
> Are these loser companies who don’t want to cache on their end
Google couldn't be bothered to cache; I would guess this before anything else.
https://drewdevault.com/blog/Google-has-been-DDoSing-sourceh...
2 replies →
There are a lot of urls for the bots to follow. Yesterday's logs had 2.2M unique URLs out of 2.7M requests (more unique than I was expecting!)
Typical traffic:
(I've moved and kept the fuzz corpus now, but other viewvc repos were suffering similarly)
I have compiled a list of vibecoders who have impressed me with their consideration for other people. Here it is:
None of the above. They're just really retarded bots. Nobody knows why they exist or what they gain from really stupid scraping.
> 1TB/month
Switch VPS providers, you can easily find higher numbers or "unlimited".
I run an ecommerce auction website on wordpress that is highly dynamic and can't be cached. Bots are a waste of cpu. I even turned off "prefetch" requests as I found that (android in particular) browsers will prefetch every auction item on the page as the user scrolls down the summary page looking at items.
My cgit instance gets hammered so hard that my CPU use is raised to 100% and the entire server grinds to a halt. I have to set a cgroups CPUQuota limit with systemd to make the server usable again.
Also, they hit so hard that nginx returns 502s and times out for real user half of the time, and when the site does load it takes a few seconds to load.
This is all on a high performance single core vultr server, not the cheapest vps but not the fastest, but should be plenty fast enough.
What about "our hosting bill shot up 500% from bot traffic and went down to the baseline the moment we put our site behind cloudflare or anubis"
My dedicated server comes with 2TB/month data included. Humans use up about 2% of that. If I didn't block bots I'd run out mid-month.
On a dedicated server? It must be somewhere like Korea, where bandwidth is extremely expensive? Usually, dedicated servers are already expensive enough that they throw in unlimited bandwidth for free. And when not, its usually more like 200TB/m
> WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that.
Yes we know - I still had my site knocked out a couple of times by overeager bots. They can be really abusive.
Wow, that's nuts. What sort of site was this? Mine don't get anything near this level of attention!
Just a personal blog, and I have to admit it had a misconfiguration that was my fault - nonexistent urls would redirect to the index. Still, dumb scraper bots would 1) not realize there was a redirect loop and kept trying every possible url, and 2) they were doing so aggressively enough that it brought the site down a couple of times. A well-behaved crawler would have slowed down and noticed the loop, never had issues with crawlers for years until the advent of ai.
One of my issues had to do with pure bandwidth, dozens of bots were requesting the same image over and over again adding random crap to the query. My server was being slow for me during crawling peaks.
> An actual proper static site would likely 10x that.
A lot of the stuff I host isn't static. Any non-static content also gets overloaded pretty quickly. Hosting something like a Git server can turn pretty resource intensive if shitty bots start diffing every commit with every other commit.
I'm generally lazy when it comes to blocking IPs, but when I do run into an issue, I just blanket block IP ranges. When that doesn't work, ASNs get blocked. So far I haven't had to take most measures mentioned here but I probably will once the broken bots start coming from botnets using residential IP addresses.
Egress costs money.
Hundreds of fake applicants to a position