Comment by solaris2007
12 days ago
Make the data available through bit-torrent and IPFS. Redirect IPs that make excessive requests to response only kilobytes in size "use the torrents and IPFS".
As an SRE, the only legitimate concern here could be the bandwidth costs. But QoS tuning should solve that too.
Supposedly technical people crying out for a journalist to help them is super lame. Everything about this looks super lame.
That data is already available. Including torrents.
https://planet.openstreetmap.org/
Perfect. Now all they need to do is set up the redirect.
Every bot is doing something on behalf of a human. Now that LLMs can churn out half-assed bot scripts every "look I installed Arch Linux and ohmyzsh" script kiddie has bots too.
Bots aren't going anywhere.
"Use the web the way it was over 10 years ago plox" isn't going to do it.
Disclosure: I am part of the OpenStreetMap mostly-volunteer sysadmin team fighting this.
The scrapers try hard to make themselves look like valid browsers, sending requests via residential IP addresses (400,000+ IPs at last count).
I reached out to journalists because despite strong technical measures, the abuse will not go away on its own.