Comment by mfost
2 days ago
I thought the point (which the article misses) is that a token gives you an identity, and an identity can be tracked and rate limited.
So a crawlers that goes very ethically and does very little strain on the server should indeed be able to crawl for a whole week on a cheap compute, one that hammers the server hard will not.
Sure but it's really cheap to mint new identities, each node on their scrapping cluster can mint hundreds of thousands of tokens per second.
Provisioning new ips is probably more costly than calculating the tokens, at least with the default difficulty setting.