← Back to context

Comment by forinti

8 days ago

My logs tell me that Google ignores my robots.txt.

Isn’t it somewhat likely that a lot of shady crawlers pretend to be a google bot with their user agent?

  • If you check the ips they belong to Google.

    Plus if you run adsense google with ignore crawler rules and visit the page from google ips and from some shady ip. Wonder if it is the same for sites using Analytics.

  • Why you still have the idea in your head that they play by the rules. With the current administration they have been empowered to extract maximum value from us.

    In the early days of smartphone use, Google and Facebook uploaded contact lists of every single smartphone user to their servers.

I was going to say, this is an absolute win for everyone with a personal site they have been trying to prevent Google from crawling if true.