Comment by stavros
8 hours ago
You really do need every single site, as search is a long tail problem. All the interesting stuff is in the fringes, if you only have a few big sites you'll have a search engine of spam.
8 hours ago
You really do need every single site, as search is a long tail problem. All the interesting stuff is in the fringes, if you only have a few big sites you'll have a search engine of spam.
I think that is only needed for a small subset of queries. Seriously think of the last time you did a search and went to a fringe site as opposed to a well known brand or social media. Ranking quality is much more important than coverage over the whole internet.
> Seriously think of the last time you did a search and went to a fringe site as opposed to a well known brand or social media.
Oh, almost never. That's exactly why search sucks now.