Comment by WmWsjA6B29B4nfk
8 days ago
Google docs are pretty clear (https://developers.google.com/crawling/docs/robots-txt/robot...):
> Google's crawlers treat all 4xx errors, except 429, as if a valid robots.txt file didn't exist. This means that Google assumes that there are no crawl restrictions.
This is a better source than a random SEO dude with a channel full of AI-generated videos.
Not entirely unlikely this is just a bug on Google's end.
It's fairly common for there to be a very long and circuitous route between cause and effect in search, so a bug like this can sometimes be difficult to identify until people start making blog posts about it.
It seems that this is not happening and even the guy who wrote the article mentions it:
> I don't have a robots.txt right now. It hasn't been there in a long time. Google still shows two results when I search for files on my site though:
The source that he links to is another indian spam channel we've seen a thousand times on YouTube
It does seem unlikely that Google would have a big in basic behavior of its crawler after ~27 years.
Google Adsense docs says that ads.txt is not mandatory and yet I remember having no ads displayed on my website until I added one.
Yeah I thought I got a notification saying to add it for an existing site but it still seemed optional the last time I created a new site?
Indeed. “Unreachable” is very different than “not found”.
[dead]