Comment by protomyth
16 years ago
It would actually be interesting to have a browser plug-in that shows you if a page you are on is excluded in the robots.txt file or which agents are specifically excluded/allowed to scan the page. Might make for some interesting viewing on sites like Facebook.
Yes, I've been thinking for a while about building an extension that shows you stuff like:
IP, location, host of the site you are viewing whois details for the site caching rules robots txt rules for the page page rank blah, etc
Probably as a dropdown info panel.
I'm surprised if this doesn't already exist, though.