Comment by xnx
11 hours ago
I realized I couldn't think of any crawler worth allowing but Googlebot. Put this in my robots.txt:
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: *
Disallow: /
11 hours ago
I realized I couldn't think of any crawler worth allowing but Googlebot. Put this in my robots.txt:
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: *
Disallow: /
No comments yet
Contribute on Hacker News ↗