← Back to context Comment by jppope 2 months ago What about for their LLM products? We know that OpenAi does not respect the robots.txt file 2 comments jppope Reply xnx 2 months ago Google uses the same crawler and robots.txt file for training data. inkysigma 2 months ago It's actually a different crawler for training data: Googlebot-extended so you can exclude yourself from the training data though not the search summaries.
xnx 2 months ago Google uses the same crawler and robots.txt file for training data. inkysigma 2 months ago It's actually a different crawler for training data: Googlebot-extended so you can exclude yourself from the training data though not the search summaries.
inkysigma 2 months ago It's actually a different crawler for training data: Googlebot-extended so you can exclude yourself from the training data though not the search summaries.
Google uses the same crawler and robots.txt file for training data.
It's actually a different crawler for training data: Googlebot-extended so you can exclude yourself from the training data though not the search summaries.