Comment by keane

The <meta name="robots"> tag and robots.txt serve different roles: robots.txt controls crawling, while the robots meta tag influences indexing and other behavior. https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/...

I wonder how archive.org_bot behaves when <meta name="robots" content="noindex, noarchive, nocache" /> is present.