← Back to context

Comment by computomatic

6 hours ago

A wiki with only 11 pages?

Perhaps they will investigate why 541,000 pages aren’t being indexed. In my experience, Google provides adequate tools for identifying and resolving indexing issues.

Google won’t serve pages it hasn’t indexed. Seems they left a lot of relevant details out of that tweet.

Edit: and the most likely answer would be that their current robots.txt disallows virtually all indexing. I’m no SEO expert but entries like this seem like footguns:

   User-agent: Google-Extended
   Disallow: /

Edit 2: there’s more info in the full thread but that was only viewable via the xcancel link someone else shared (despite having the X app installed - deeplinks don’t work today). A helpful example of why X is not the best platform for sharing multi-post threads. Seems robots.txt was considered but ruled out.

In the first image you can see how indexed pages go from 40k+ to 11 in the matter of days. Further down the thread I show how 114k+ pages are marked as “crawled but not index” and we can’t understand why. The rest is stuff that is (correctly) blocked by robots.txt.