Comment by SoftTalker

This is entirely because "we" insist on search being free. This means Google needs to find other ways to pay for it, which creates a different set of incentives.

If we somehow paid directly for search, then Google's incentives would be to make search good so that we'd be happy customers and come back again, rather than find devious ways to show us more ads.

Most people put up with the current search experience because they'd rather have "free" than "good" and we see this attitude in all sorts of other markets as well, where we pay for cheap products that fail over and over rather than paying once (but more) for something good, or we trade our personal information and privacy for a discount.