Comment by thayne
19 hours ago
You need to be careful with blocking based on IP. IP blocks get reassigned every once in awhile, and you can end up blocking the wrong people. I've seen multiple cases where a block of IPs was blocked either because based on geography, or being a data center, then the block changed hands to a residential ISP.
Blocking http 1.1 has a higher risk of blocking real users if they are using old browsers.
> One class of bots always claims to come from the root page of google. Obviously this is a lie.
This isn't just bots. Some browsers won't include the full url for cross-origin requests, so that is the referrer if the request came from google, so you probably don't want to block that unless you don't want any traffic from google search.
For what it's worth I block all of Google's ASN's entirely. Just me personally for my silly hobby and tinkering sites, I have not received any good traffic from them as of late. Even their search has gone to pot in my opinion.
I will keep the blocking of http/1.1 at the risk of blocking ancient browsers and API tools. That's just too easy for people to update these days and they are not missing much by not visiting cranky ol' me. If we were talking about proprietary code running in an old financial system I would totally get it, but the public internet can and should update for their own sake.
Any request coming to me claiming to be from Google I know is a lie as I have been blocking Google for a fairly long time. If anything Google would say I am malicious for blocking them, or at least they used to. I cycle by blog through many random domains to break any associations and related snapshots to disrupt birds of a feather associations. I try to control how people discover my silly ramblings.
An unfortunately large amount of people unknowingly having their network resold for residential vpns too.
Blocking http1.1 is reasonable imo, its been over a decade since just about every browser has supported it. Anyone on a browser that old has most the mainstream web broken to them at this point, a broken (personal) site wouldnt be an exception to them, but the norm.
I blocked AWS incoming traffic a few years ago. I wrote a blog post about it. I get maybe dozens of actual visitors a week; I got around 15,000 real people views of the post in about three months. Amazing how quickly things like that drop below the memory horizon for people.
I also got a free pentest. I've expanded since. Takeaways: my mitigations and pipeline are solid. If the people on HN pleading for the poor LLM users and google search have any relationship with the people providing the free pentest... I mean, they don't? Srsly.
As for blocking IPs... the survival pressure has forced 90% of the bot traffic to VPNs. Man, they're lighting up the VPN endpoints like christmas trees. Little points of light, very pretty. Thanks for the fish. I could make feeds of the one-shot wonders available, my infra is solid, you'd need to be properly vetted and I'd need to approve the use case.
This is fun.
Author has indicated he's perfectly fine with blocking all sorts of human users. I wouldn't take any of his advice.