Comment by estimator7292 20 days ago What I'm hearing is that if I tweak robots.txt I can exclude my site from google? Excellent news! 8 comments estimator7292 Reply sofixa 20 days ago This is literally the point of robots.txt. It was created to allow site owners to configure how and which parts of their website can be scraped by what bot, and all the "decent" ones (Google, Bing) respect it. bflesch 20 days ago Spoiler: They don't. edwinjm 20 days ago Google, Bing do aendruk 20 days ago Was that not possible before? dazc 20 days ago no-index is a thing. tremon 20 days ago no-index is per individual page, not for an entire domain, IIUC? snowwrestler 20 days ago You can pretty easily send no-index for an entire site by configuring it as a site-wide HTTP header.
sofixa 20 days ago This is literally the point of robots.txt. It was created to allow site owners to configure how and which parts of their website can be scraped by what bot, and all the "decent" ones (Google, Bing) respect it. bflesch 20 days ago Spoiler: They don't. edwinjm 20 days ago Google, Bing do
dazc 20 days ago no-index is a thing. tremon 20 days ago no-index is per individual page, not for an entire domain, IIUC? snowwrestler 20 days ago You can pretty easily send no-index for an entire site by configuring it as a site-wide HTTP header.
tremon 20 days ago no-index is per individual page, not for an entire domain, IIUC? snowwrestler 20 days ago You can pretty easily send no-index for an entire site by configuring it as a site-wide HTTP header.
snowwrestler 20 days ago You can pretty easily send no-index for an entire site by configuring it as a site-wide HTTP header.
This is literally the point of robots.txt. It was created to allow site owners to configure how and which parts of their website can be scraped by what bot, and all the "decent" ones (Google, Bing) respect it.
Spoiler: They don't.
Google, Bing do
Was that not possible before?
no-index is a thing.
no-index is per individual page, not for an entire domain, IIUC?
You can pretty easily send no-index for an entire site by configuring it as a site-wide HTTP header.