Comment by jsnell
8 days ago
This was beyond predictable. The monetary cost of proof of work is several orders of magnitude too small to deter scraping (let alone higher yield abuse), and passing the challenges requires no technical finesse basically by construction.
We need to revive 402 Payment Required, clearly. If we lived in a world where we could easily set up a small trusted online balance for microtransactions that's interoperable with everyone, and where giving others a literal penny for their thoughts could allow for running up a significant bill for abusers, I'd gladly play along.
Me too. I wouldn't mind Project Xanadu style micro payments for blogs, and it'd both fix the AI scraper issue and the ads issue, and help people fund hosting costs sustainably. I think the issue is taxes and transaction fees would push the prices too high, and it'd price out people with very low income possibly. It'd also create really perverse incentives for even more tight copyright control, since your content appearing even in part on anyone else's website is then directly losing you money, so it'd destroy the public Commons even more, which would be bad. But maybe not, who knows.
Cloudflare are working on this at the moment.
https://blog.cloudflare.com/introducing-pay-per-crawl/
There's also an open specification called x402:
https://www.x402.org/x402-whitepaper.pdf
I would definitely use this to charge US$100,000 per request from any AI company to crawl my site. I would exempt 'public good' crawlers like The Internet Archive though.
If AI companies valued at billions of dollars want to slurp up my contribution to the human condition, that's my price - subject to price rises only.
Pay to visit would be great, and would force these AI companies to actually pay for their data.
For someone doing spamming that low level would work well. As their cost is determinatively low to make it work. For someone doing scraping to get data and feeding it to an AI not so much. The AI groups usually have some pretty heavy hitting hardware sitting behind it. They could even break off some hardware that is to be retired and have it munch away on it. To make it non cost effective the calculations would need to be much bigger.