Comment by Aardwolf
8 days ago
If true, this would mean more websites with genuine content from the "old" internet won't show up (since many personal websites won't have this), while more SEO-optimized content farms that of course do put up a robots.txt will...
It also fits Google's plan to create a surrogate web.
- AI was the first step (or actually, among the first five steps or so). CHECK. - Google search has already been ruined. CHECK. - Now robots.txt is used to weed out "old" websites. CHECK.
They do too much evil. But it is also our fault, because we became WAY too dependent on these mega-corporations.