Comment by senko

5 hours ago

A crawler that respects robots.txt is useless in practice since many sites only allow Googlebot and maaaybe Bing - by name.

0 comments