Comment by AnthonyMouse
2 days ago
It's nose diving due to Google putting the LLM answer box at the top of the search results. But then isn't it even worse to be blocking every search engine crawler that isn't doing that?
2 days ago
It's nose diving due to Google putting the LLM answer box at the top of the search results. But then isn't it even worse to be blocking every search engine crawler that isn't doing that?
Even companies like OpenAI segregate their training crawlers (GPTBot for general corpus training; OAI-SearchBot for their search index inclusion; and ChatGPT-User for user-driven agentic browser use, etc).
When even OpenAI is more respectful of intellectual property and website owner control than you, that's a problme.
> Even companies like OpenAI segregate their training crawlers (GPTBot for general corpus training; OAI-SearchBot for their search index inclusion; and ChatGPT-User for user-driven agentic browser use, etc).
They have three different user agent strings, anyway. The problem is obviously that you can't tell what someone does with data after you give it to them.
You also don't know that some third party isn't crawling the web with the user agent string "OAI-SearchBot" and then using the results for training or selling the data to the likes of Anthropic or OpenAI without telling them that.
In general attempting to use the user agent string for access control is not going to work and it seems like Google is being the less disingenuous party on this one by not blowing smoke.