Comment by dawnerd

What's the difference with https://github.com/TecharoHQ/anubis?

Considering sibuna it's anubis backwards.

It's a cognitive tax on all the parties involved. It works on buyers (websites that pay for it) much the way that llm's work on their users, by working up to the point at which the buyers can't discern it working.

As a user (bot or flesh) you have to jump through these hoops, and to the website operators they get fewer complaints when users (and bots) are able to contort themselves in predictable ways. Versus you'd perceive as a website operator, if your blocking worked at all, that you'd risk seeing a precipitous dropoff of bots if you did things in bespoke ways, as both bots (true positives, among dd and cf survivors) and people (false positives) get jammed.

The net effect in this model would be that such things as cloudflare and datadome take money from websites in order to block "dumb" bots and enfranchise "smart" bots to the point of creating a market for bot services. People that want to bot have to use AI or go to bot service providers.

For website operators that's a worthwhile exchange for dollars, in that dumb bots can cause a lot of harm in wasted compute. Bots are typically dead end users who won't buy ever anyway, or merely inefficient way of doing business if they scrape excessively. And many do scrape excessively.

"reform school for bots" is a good mental model. For this to work, the "reforms" have to be consistent so that bots can conform to them.

It's a peculiar equilibrium and I am not sure that it's all that stable as an equilibrium. Would be a good subject for a formal treatment, game theory and economics.

Mostly economies of scale. They have enough data to do the statistics to identify bad actors, and enough bandwidth and servers to facetank the request volume.

Would love if they did but most end up just reselling cloudflare. Example WPEngine.

Cloudflare is probably one of the most popular, reasonably priced for general public. You have fastly, akamai, cloudfront as other proxy providers and CDNs with similar capability but higher price point mostly catering to enterprise customers.

Bunny.net is another one that I find similar to Cloudflare from pricing and functionality perspective

Make your site performance fast enough that bots don't bother you.

Depends on the small business obviously. I've been in IT for 40+ years and enjoy torturing clankers; I ran a bot motel 20 years ago. These things are so badly written, and obviously they don't care because they're not truly conscious, but the meat in charge is ignorant or doesn't care. So web scraping, and the point cloud of nearly indistiguishable abuse it travels with, is pretty easily tripped by some simple technical tests and salting of content. At least mostly.[0]

I've been wondering to myself "how can I get people to go through a login process without going through a login process?" for a few months, and I've come up with a simple solution which I'll be implementing in the next few weeks.[1] Like the other technical measures it doesn't identify bots but there will be a gate with backpressure that at a point in time a given IP address has some probability (adjusted in realtime based on $things) of passing through, and through my mysterious process is then "logged in". Then that address is pretty much given a free pass as long as its behavior remains "human". Real humans may be denied access initially and have to try again, and their luck will be affected by how aggressively the bots are fighting it out at that point in time. This isn't much different than the extant reality, but it is in terms of what happens after an initial successful page load. It's an experiment, I'm hoping it's tunable so that the experience is more or less indistinguishable from the status quo; we will see.

I'd love to find a group of likeminded individuals where we can discuss some of this TLP:YELLOW or Chatham House Rules. If you are savvy enough to hunt me down and can demonstrate that you're a real human, small operator, and that you can keep things closely held I'll talk with you, if you want someone to run your mitigations that would cost money.

[0] Had an episode last week where somebody scanned approximately 1000 URLs (maybe 60% of the discoverable URLs) over six hours. 1200 addresses fetched exactly one URL, another 400 or so fetched two or three. Every one of those URLs was fetched exactly two times. Real showshoe stuff, nicely done.

[1] Firewall rules and fail2ban. I give fail2ban special powers in terms of the logs it follows and the actions it can invoke. Conntrack without conntrack (of course the firewall has conntrack, it's just not the best option for this use case).

TikTok, Instagram, X, etc. These platforms will host you for free and will distribute updates to people, including potential customers via the algorithm.

[flagged]

Not it the site can't stay up because it's constantly being hammered by bots.

Dude, I had a 1Gb of nginx logs daily then I blocked them and it dropped to 10mb weakly.