Comment by za_creature
16 hours ago
Unfortunately, geofencing is becoming more and more common specifically because of the extra bot traffic.
A shop selling goods in $COUNTRY has very little to gain from allowing arbitrary users from any other country. Allowing bots has always been a tradeoff between the potential benefits and the impact on infrastructure, which is why it's reasonable to identify yourself as a bot for the purpose of rate limiting purposes.
A journalist anonymously using a crawler is indistinguishable from a bad actor trying to collect data for its own purposes, some of which may actually be against the interests of the company in question. In particular, a blog or news publication offering knowledge in exchange for ad revenue stands to disappear because that same knowledge can be compressed in a LLM and offered to the end user directly, completely depriving the source of any revenue.
> Unfortunately, geofencing is becoming more and more common specifically because of the extra bot traffic.
How is geofencing supposed to help? The crawler operated by technical staff in $foreign_country just switches to compromised smart TVs in your own country and you ultimately end up blocking primarily real humans who don't all know how to change their apparent geolocation.
> A journalist anonymously using a crawler is indistinguishable from a bad actor trying to collect data for its own purposes, some of which may actually be against the interests of the company in question.
There is a major way to distinguish them. If you pass a law requiring them to do something, journalists from major publications have legal departments who then order them to follow the law (and then can't do their journalism), whereas the bad actor is either not interested in following laws or is not in your jurisdiction to be subject to them to begin with.
> The crawler operated by technical staff in $foreign_country just switches to compromised smart TVs in your own country
That costs the crawler money, not many do it and is something that can be fixed over time
banning country and AS numbers for cloud companies is pretty effective in my experience
> That costs the crawler money
They're for-profit operations. They have money, and it doesn't cost that much. Proof of work is significantly more expensive (but then you put delays on even the users in your own region).
> banning country and AS numbers for cloud companies is pretty effective in my experience
It's the sort of thing that works when the first person initially rolls it out because the crawlers only bother to do something about it once a threshold percentage of people are doing it, at which point they use the workaround.
And then you have people insisting that it works even though it only used to work, and meanwhile you're still blocking actual humans.
1 reply →
Yep doing that (actually just countries) eliminated card testing on an e commerce site I maintain.
From hundreds a day, to zero.
The compromised smart tvs in your country were manufactured by someone that could be held liable for operating a "stealth" bot network, or for allowing someone to do so through negligence.
As for journalists, nothing is stopping their legal department from petitioning a judge to issue a temporarily exemption from the law for a specific period of time under a specific written purpose.
> As for journalists, nothing is stopping their legal department
and this is the point.
so people without a 'legal department' are shut off.
1 reply →
> A shop selling goods in $COUNTRY has very little to gain from allowing arbitrary users from any other country.
Eh?
a) Window shopping for the next time the out-of-country visitor is in the country.
b) An out-of-country visitor can contract someone who's in the country to purchase things from the shop and ship those things to them.
c) Queries and requests from out-of-country visitors can cause the shop to notice that setting things up so that they can ship internationally actually would be a money-making thing for them to do.
I do agree that the obscenely antisocial bots run for the benefit of the LLM Manufacturing Industry make it very hard for small-time operators to keep their websites up and running at any reasonable cost. Hopefully the major LLM Manufacturers will fail to secure enough funding over the next few months, their VC-backed primary customers will then have their money faucets shut off, and the attack bots will reduce down to a dull roar because folks have stopped paying "infinity" dollars for them.
All of the above fall under "very little". We're talking in the ballpark of 5% extra sales max, at the expense of double the infrastructure costs.
They're also stealing your curated dataset: if you take the time to correctly label your products, add specifications and manuals because you want this to be a competitive advantage, your work was just re-appropriated for the benefit of humanity. But primarily for your larger competitors willing to pay for it.