Comment by emayljames
5 years ago
Yes, I have noticed that creating a brand new dev domain with crawler blocking norobots file, it is not found on any search on Google, until I open the dev url in Chrome, then bam! watch as their crawler starts trying to search through the site just from opening the url in Chrome.
This is why I never use Chrome. They scrape the Google Safe Browsing sent from chrome browsers and just do not care about privacy.
Maybe it's from search suggestion API? Anyway, I turn that off as soon as I create a new browser profile, along with the safe browsing list and automatic search when I type unrecognized URL. When I want to search I use search input of the browser. (ctrl+k) URL bar is for URLs only.
You realize that robots.txt is an "on your honor" system and that any one can write a script that doesn't look at robots.txt and post anything they find to the internet and that therefore other sites could find your site via 3rd party data.
Chrome does not do what you claim it does
I have trialed this several times, not using chrome and everytime I then use it, the site can be found on google. Remember, these sites are completely unlinked and fresh URLs. So, yeah it really does..