Comment by crazygringo

This is interesting and unexpected if true.

My only thought is that virtually all "serious" sites tend to have robots.txt, and so not having it indicates a high likelihood of spam.