Comment by azeemba
11 hours ago
Crawler, AI or not, cannot afford to pay per visit. The entire model of crawling works because the incremental cost of each crawl is so low. Even fractions of a penny would be prohibitive.
11 hours ago
Crawler, AI or not, cannot afford to pay per visit. The entire model of crawling works because the incremental cost of each crawl is so low. Even fractions of a penny would be prohibitive.
As it should be. Bots are still user agents much like browsers
If you're paying per token for AI, you can also pay a smaller amount to use the Web.