Comment by mystraline
6 hours ago
IP blocking Asia took my abusive scans down 95%.
I also do not have a robots.txt so google doesnt index.
Got some scanners who left a message how to index or dei dex, but was like 3 lines total in my log (thats not abusive).
But yeah, blocking the whole of Asia stopped soooo much of the net-shit.
> I also do not have a robots.txt so google doesnt index.
That doesn't sound right. I don't have robots.txt too but Google indexes everything for me.
https://news.ycombinator.com/item?id=46681454
I think this is a recent change.
All the comments there seem to suggest that there has been no change and that robots.txt isn't required.
How did you block Asia, cloudflare or something else?