Comment by AkshatM

The blog post covers this. The announcement also drops relying on spoofable user agents for crawler identification and requires crawlers to voluntarily identify themselves via RFC 9421 cryptographic message signatures to get access: https://blog.cloudflare.com/introducing-pay-per-crawl/#payme...

There are likely incentives for AI companies to try to simulate human users as much as possible, but the value proposition here is that CF is so good at identifying and stopping those that signing a request becomes the path of least resistance.

Disclosure: I am on the team that wrote the RFC 9421 message signature implementation at Cloudflare and its use in the pay per crawl project. A separate blog post went out here: https://blog.cloudflare.com/verified-bots-with-cryptography/