Comment by bensyverson 1 month ago Looks about as effective as Robots.txt 5 comments bensyverson Reply cgannett 1 month ago Robots.txt became 0% effective eventually. But this? With the way LLMs work? You never know. dguest 1 month ago Claude seems to follow robots.txt by default. Actually at my organization our theory is that this is why no one is finding our public results any more. warkdarrior 1 month ago I am adding it to my instruction-following training data, as a negative sample. Den_VR 1 month ago People still rail against Robots.txt crimes, to the point of self destructing all their own content.
cgannett 1 month ago Robots.txt became 0% effective eventually. But this? With the way LLMs work? You never know. dguest 1 month ago Claude seems to follow robots.txt by default. Actually at my organization our theory is that this is why no one is finding our public results any more. warkdarrior 1 month ago I am adding it to my instruction-following training data, as a negative sample. Den_VR 1 month ago People still rail against Robots.txt crimes, to the point of self destructing all their own content.
dguest 1 month ago Claude seems to follow robots.txt by default. Actually at my organization our theory is that this is why no one is finding our public results any more.
warkdarrior 1 month ago I am adding it to my instruction-following training data, as a negative sample.
Den_VR 1 month ago People still rail against Robots.txt crimes, to the point of self destructing all their own content.
Robots.txt became 0% effective eventually. But this? With the way LLMs work? You never know.
Claude seems to follow robots.txt by default. Actually at my organization our theory is that this is why no one is finding our public results any more.
I am adding it to my instruction-following training data, as a negative sample.
People still rail against Robots.txt crimes, to the point of self destructing all their own content.