Comment by mysteria

7 months ago

Even before AI was a thing some websites would deny all crawlers in robots.txt except for the Googlebot for the same reason.

0 comments