Comment by keane
2 days ago
The <meta name="robots"> tag and robots.txt serve different roles: robots.txt controls crawling, while the robots meta tag influences indexing and other behavior. https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/...
I wonder how archive.org_bot behaves when <meta name="robots" content="noindex, noarchive, nocache" /> is present.
The person above those is complaining about entries in their logs from bots. A robot can't read a tag without first reading the document. So sure, if they're a good bot they might not store the results but the server's logs will still show the bot's GET request.