Comment by _V_

Yeah, I have correctly set up robots.txt - if they won't respect that, F them. Bandwidth is not free and I don't mind giving it out to individuals, but I'm not feeding multi-billion dollar companies.