Comment by marginalia_nu
7 hours ago
To be honest it's probably just jank on Google's end.
There's a lot of delayed cause and effect in search, and it's much easier to make a minor mistake that excludes 0.1% of websites from crawling or indexing than it is to detect that it's happened except from affected websites telling you about it.
Like in marginalia I've had a bug that affected websites in the condition that if the root path didn't support HEAD, but did support GET with a `Range` header, and it correctly responded with a HTTP 206, then the website wouldn't be indexed because some code that was testing the root document for issues as an initial probe handled that as an error state. Most websites that support range requests also support HEAD (as this usually means the document isn't generated). Except a handful of Caddy-based configurations, about 0.3% of servers.
Or just some AI flagging it as some sort of content they don't want to show. There's no way they can be perfect at that.
from 511k indexed paged to just 11? that is some serious jank
From the perspective of a web search engine indexing tens to hundreds of billions of documents 500k docs is not very noticeable.