Comment by estimator7292 8 days ago What I'm hearing is that if I tweak robots.txt I can exclude my site from google? Excellent news! 8 comments estimator7292 Reply sofixa 8 days ago This is literally the point of robots.txt. It was created to allow site owners to configure how and which parts of their website can be scraped by what bot, and all the "decent" ones (Google, Bing) respect it. bflesch 8 days ago Spoiler: They don't. edwinjm 8 days ago Google, Bing do aendruk 8 days ago Was that not possible before? dazc 8 days ago no-index is a thing. tremon 8 days ago no-index is per individual page, not for an entire domain, IIUC? snowwrestler 8 days ago You can pretty easily send no-index for an entire site by configuring it as a site-wide HTTP header.
sofixa 8 days ago This is literally the point of robots.txt. It was created to allow site owners to configure how and which parts of their website can be scraped by what bot, and all the "decent" ones (Google, Bing) respect it. bflesch 8 days ago Spoiler: They don't. edwinjm 8 days ago Google, Bing do
dazc 8 days ago no-index is a thing. tremon 8 days ago no-index is per individual page, not for an entire domain, IIUC? snowwrestler 8 days ago You can pretty easily send no-index for an entire site by configuring it as a site-wide HTTP header.
tremon 8 days ago no-index is per individual page, not for an entire domain, IIUC? snowwrestler 8 days ago You can pretty easily send no-index for an entire site by configuring it as a site-wide HTTP header.
snowwrestler 8 days ago You can pretty easily send no-index for an entire site by configuring it as a site-wide HTTP header.
This is literally the point of robots.txt. It was created to allow site owners to configure how and which parts of their website can be scraped by what bot, and all the "decent" ones (Google, Bing) respect it.
Spoiler: They don't.
Google, Bing do
Was that not possible before?
no-index is a thing.
no-index is per individual page, not for an entire domain, IIUC?
You can pretty easily send no-index for an entire site by configuring it as a site-wide HTTP header.