← Back to context Comment by estimator7292 1 month ago What I'm hearing is that if I tweak robots.txt I can exclude my site from google? Excellent news! 8 comments estimator7292 Reply sofixa 1 month ago This is literally the point of robots.txt. It was created to allow site owners to configure how and which parts of their website can be scraped by what bot, and all the "decent" ones (Google, Bing) respect it. bflesch 1 month ago Spoiler: They don't. edwinjm 1 month ago Google, Bing do aendruk 1 month ago Was that not possible before? dazc 1 month ago no-index is a thing. tremon 1 month ago no-index is per individual page, not for an entire domain, IIUC? snowwrestler 1 month ago You can pretty easily send no-index for an entire site by configuring it as a site-wide HTTP header.
sofixa 1 month ago This is literally the point of robots.txt. It was created to allow site owners to configure how and which parts of their website can be scraped by what bot, and all the "decent" ones (Google, Bing) respect it. bflesch 1 month ago Spoiler: They don't. edwinjm 1 month ago Google, Bing do
dazc 1 month ago no-index is a thing. tremon 1 month ago no-index is per individual page, not for an entire domain, IIUC? snowwrestler 1 month ago You can pretty easily send no-index for an entire site by configuring it as a site-wide HTTP header.
tremon 1 month ago no-index is per individual page, not for an entire domain, IIUC? snowwrestler 1 month ago You can pretty easily send no-index for an entire site by configuring it as a site-wide HTTP header.
snowwrestler 1 month ago You can pretty easily send no-index for an entire site by configuring it as a site-wide HTTP header.
This is literally the point of robots.txt. It was created to allow site owners to configure how and which parts of their website can be scraped by what bot, and all the "decent" ones (Google, Bing) respect it.
Spoiler: They don't.
Google, Bing do
Was that not possible before?
no-index is a thing.
no-index is per individual page, not for an entire domain, IIUC?
You can pretty easily send no-index for an entire site by configuring it as a site-wide HTTP header.