Comment by account42

> Microsoft can throw around $60M, and Bing is used by most of the "alternative" search engines.

That's a symptom of the issue, not a solution. Bing is used because having your own crawler is infeasible, partially because you will be literally blocked in many cases.