Comment by eli
4 hours ago
Strictly speaking no, that doesn’t prevent crawling - at the least Googlebot has to fetch the page to see the meta tag or the robots.txt to see what’s allowed, and it will periodically recheck for changes.
It doesn’t even prevent indexing. If a page is linked from elsewhere, Google will show it in search results even if noindex’d.
And why does Google get to set these rules on my site anyway? I didn’t agree to them.
No comments yet
Contribute on Hacker News ↗