Comment by misswaterfairy

Cloudflare are working on this at the moment.

https://blog.cloudflare.com/introducing-pay-per-crawl/

There's also an open specification called x402:

https://www.x402.org/x402-whitepaper.pdf

I would definitely use this to charge US$100,000 per request from any AI company to crawl my site. I would exempt 'public good' crawlers like The Internet Archive though.

If AI companies valued at billions of dollars want to slurp up my contribution to the human condition, that's my price - subject to price rises only.