Comment by madibo3156

However, so-called AI crawlers are not the same as crawlers of yore. They hit live pages every time a user prompt triggers a web search.

This Web Search API, unlike an AI crawler, only fetches periodically. It feels like a step in the right direction for managing resource strain across the internet. If only the LLM giants could do something similar.