Comment by markerz

Yep. Same for most of Asia too.

Cloudflare's filters are basically straight up racist.

I have stopped using so many sites due to their use of Cloudflare.

People hate collective punishment because it works so well.

I have a growing Mastodon thread of this shit: https://mastodon.social/@grishka/111934602844613193

It's of course trivially bypassable with a VPN, but getting a 403 for an innocent get request of a public resource makes me angry every time nonetheless.

The difference between politics and diplomacy is that you can survive in politics without resorting to collective punishment.

unrelated: USSR might have been 2nd world. Russia is 3rd world (since 1991) -- banana republic

If you live in a region which it is economically acceptable to ignore the existence of (I do), you sometimes get blocked by website r̶a̶c̶k̶e̶t̶ protection for no reason at all, simply because some "AI" model saw a request coming from an unusual place.

Sometimes it doesn’t even give you a Captcha.

I have come across some websites that block me using Cloudflare with no way of solving it. I’m not sure why, I’m in a large first-world country, I tried a stock iPhone and a stock Windows PC, no VPN or anything.

That’s just no way to know.

If it clears you at all. I accidentally set a user agent switcher on for every site instead of the one I needed it for, and Cloudflare would give me an infinite loop of challenges. At least turning it off let me use the Internet again.

The risk is not to the company using Cloudflare; the risk is to any legitimate individual who Cloudflare decides is a bot. Hopefully their detection is accurate because a false positive would cause great difficulties for the individual.

And not just a login but soon probably also the real verified identity tied to it. The internet is becoming a worse place than the real world.

Self plug, but I made this to deal with bots on my site: https://marcusb.org/hacks/quixotic.html. It is a simple markov generator to obfuscate content (static-site friendly, no server-side dynamic generation required) and an optional link-maze to send incorrigible bots to 100% markov-generated non-sense (requires a server-side component.)

> You could run all of your content through an LLM to create a twisted and purposely factually incorrect rendition of your data. Forward all AI bots to the junk copy.

> Everyone should start doing this. Once the AI companies engorge themselves on enough garbage and start to see a negative impact to their own products, they'll stop running up your traffic bills.

I agree, and not just to discourage them running up traffic bills. The end-state of what they hope to build is very likely to be extremely for most regular people [1], so we shouldn't cooperate in building it.

[1] And I mean end state. I don't care how much value you say you get from some AI coding assistant today, the end state is your employer happily gets to fire you and replace you with an evolved version of the assistant at a fraction of your salary. The goal is to eliminate the cost that is our livelihoods. And if we're lucky, in exchange we'll get a much reduced basic income sufficient to count the rest of our days from a dense housing project filled with cheap minimum-quality goods and a machine to talk to if we're sad.

Their problem is they can’t detect which are bots in the first place. If they could, they’d block them.

> Everyone should start doing this. Once the AI companies engorge themselves on enough garbage and start to see a negative impact to their own products, they'll stop running up your traffic bills

Or just wait for after the AI flood has peaked & most easily scrapable content has been AI generated (or at least modified).

We should seriously start discussing the future of the public web & how to not leave it to big tech before it's too late. It's a small part of something i am working on, but not central. So i haven't spend enough time to have great answers. If anyone reading this seriously cares, i am waiting desperately to exchange thoughts & approaches on this.

You will be burning through thousands of dollars worth of compute to do that.

I don't understand this. You don't have routes your users might need in robots.txt? This article is about bots accessing resources that other might use.

How can I implement this?

Whatever M$ was doing still baffles me. I still have several azure ranges in my blocklist because whatever this was appeared to change strategie once I implemented a ban method.

If it doesn't say it's a bot and it doesn't come from a corporate IP it doesn't mean it's NOT a bot and not run by some "AI" company.

Don't worry, the governments are perfectly capable of coming up with excuses all on their own.

There’s a fine line between scrapping and DDOS’ing I’m sure.

Just because you manufacture chemicals doesn’t mean you can legally dump your toxic waste anywhere you want (well shouldn’t be allowed to at least).

You also shouldn’t be able to set your crawlers causing sites to fail.

If I make a physical robot and it runs someone over, I'm still liable, even though it was a delivery robot, not a running over people robot.

If a bot sends so many requests that a site completely collapses, the owner is liable, even though it was a scraping bot and not a denial of service bot.

Then you can feed them deliberately poisoned data.

Send all of your pages through an adversarial LLM to pollute and twist the meaning of the underlying data.

Unless they're a big company in which case they can DMCA anything they want, and they get the benefit of the doubt.

For reference:

> If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really).

It's really absurd that they seem to think this is acceptable.

Edit history of a wiki sounds much more interesting than the current snapshot if you want to train a model.

Or invisible text to humans about such topics.

[flagged]

> releases their models

Some of them, and initially only by accident. And without the ingredients to create your own.

Meta is trying to kill OpenAI and any new FAANG contenders. They'll commoditize their complement until the earth is thoroughly salted, and emerge as one of the leading players in the space due to their data, talent, and platform incumbency.

They're one of the distribution networks for AI, so they're going to win even by just treading water.

I'm glad Meta is releasing models, but don't ascribe their position as one entirely motivated by good will. They want to win.

The Banality of Evil.

Everyone has to pay bills, and satisfy the boss.

DDOS is different from crashing.

And I doubt Facebook implemented something that actually saturates the network, usually a scraper implements a limit on concurrent connections and often also a delay between connections (e.g. max 10 concurrent, 100ms delay).

Chances are the website operator implemented a webserver with terrible RAM efficiency that runs out of RAM and crashes after 10 concurrent requests, or that saturates the CPU from simple requests, or something like that.

I suspect if the tables were turned and someone managed to crash FB consistently they might not take too kindly to that.

I wouldn't expect it to crash in any case, but I'd generally expect that even an n100 minipc should bottleneck on the network long before you manage to saturate CPU/RAM (maybe if you had 10Gbit you could do it). The linked post indicates they're getting ~2 requests/second from bots, which might as well be zero. Even low powered modern hardware can do thousands to tens of thousands.

Anyone can send 10k concurrent requests with no more than their mobile phone.

Read the article, the bots change their User Agent to an innocuous one when they start being blocked.

And having to use Cloudflare is just as bad for the internet as a whole as bots routinely eating up all available resources.