Comment by fc417fc802
2 days ago
You are asserting that they don't work without explanation or evidence. Meanwhile it isn't clear why they wouldn't and indeed they appear to accomplish the stated goal of severely rate limiting scrapers.
2 days ago
You are asserting that they don't work without explanation or evidence. Meanwhile it isn't clear why they wouldn't and indeed they appear to accomplish the stated goal of severely rate limiting scrapers.
IIRC there was a blog post here on HN recently explaining in detail why Anubis doesn't keep out AI bots.
Yes the default config can be trivially bypassed. When it becomes a problem the UA exceptions can be removed. It's no different than how difficulty can be scaled up and down as circumstance demands.
> You are asserting that they don't work without explanation or evidence.
Oh, no, that isn't how it works — you made a claim first (albeit indirectly), with no evidence. It's on you
Evidence: they solved many people's problems
That's not how it works. There are cases where established norms or common sense dictate a certain default assumption. This is one of them.
Still, I'll humor your absurd request by reminding you of the many success stories that have repeatedly made the front page of HN.
Beyond that we have cryptocurrencies. If you have knowledge of a generalized solution for defeating PoW schemes then why are you posting here instead of making yourself a billionaire?
Surely I don't need to explain how headless browsers work, bot proxies and distributed crawlers to you do I? If you haven't experienced this first-hand, I can understand. Maybe it warrants a blog post. But I can assure you, these don't work at scale, they might only stop the "less sophisticated" bots and maybe (just maybe) some unwanted spam.
What do the browsers being headless or the traffic proxied have to do with anything? Each client still has to solve a PoW challenge regardless. That mitigates the (possibly unintentional) DoS by making it exorbitantly expensive.