Comment by freetonik
21 hours ago
>Search crawlers have the goal of directing people towards the websites they crawl. They have a symbiotic relationship, so they put in (some) effort not to blow websites out of the water with their crawling, because a website that's offline is useless for your search index.
Ultimately not true. Google started showing pre-parsed "quick cards" instead of links a long time ago. The incentives of ad-driven search engines are to keep the visitors on the search engine rather than direct them to the source.
> The incentives of ad-driven search engines are to keep the visitors on the search engine rather than direct them to the source.
It's more complicated than that. Google's incentives are to keep the visitors on the search engine only if the search result doesn't have Google ads. Though it's ultimately self-defeating I think, and the reason for their decline in perceived quality. If you go back to the backrub whitepaper from 1998, you'll find Brin and Page outlining this exact perverse incentive as the reason why their competitors sucked.