Comment by mystraline
25 days ago
IP blocking Asia took my abusive scans down 95%.
I also do not have a robots.txt so google doesnt index.
Got some scanners who left a message how to index or dei dex, but was like 3 lines total in my log (thats not abusive).
But yeah, blocking the whole of Asia stopped soooo much of the net-shit.
> I also do not have a robots.txt so google doesnt index.
That doesn't sound right. I don't have robots.txt too but Google indexes everything for me.
https://news.ycombinator.com/item?id=46681454
I think this is a recent change.
All the comments there seem to suggest that there has been no change and that robots.txt isn't required.
How did you block Asia, cloudflare or something else?
You can block at your gateway/router. Lots of places have country IP ranges[1], and there are even more or less frequently updated lists of 'malicious' IP ranges[2]. Some gateway providers include 'block by country' and/or 'download blocklists automatically' as a feature.
[1] e.g. https://github.com/ipverse/geo-ip-blocks
[2] e.g. https://github.com/bitwire-it/ipblocklist
You can download weekly IP blocks of regions.
I import them into iptables and wholesale block them all.
I dont deal with eastdakota's pile of shit.
If you block the rest you get to 100%.