Comment by baggachipz
18 hours ago
A classic case of climbing the wall, and pulling the ladder up afterward. Others try to build their own ladder, and Google uses their deep pockets and political influence to knock the ladder over before it reaches the top.
Why does Google even need to know about your ladder? Build the bot, scale it up, save all the data, then release. You can now remove the ladder and obey robots.txt just like G. Just like G, once you have the data, you have the data.
Why would you tell G that you are doing something? Why tell a competitor your plans at all? Just launch your product when the product is ready. I know that's anathema to SV startup logic, but in this case it's good business
Running the bot nowadays is hard, because a lot of sites will now block you - not just by asking nicely via robots.txt, but by checking your actual source IP. Once they see it's not Google, they send you a 403.
Cloudflare’s ubiquity makes bootstrapping a search index via crawler virtually impossible, but what about data sources like Common Crawl?
Cost, presumably. From the article:
> Microsoft spent roughly $100 billion over 20 years on Bing and still holds single-digit share. If Microsoft cannot close the gap, no startup can do it alone.
Wouldn't it be nice if Microsoft opened the bing index for all.
2 replies →
Now that you mention it...
It's odd that Microsoft hasn't aggressively pushed for "openness". That's in the usual playbook for attacking a market leader.
(And then pull up the ladder once you've become king of hill.)
Microsoft will probably never topple Google, absent anti-monopolistic enforcement. But they can certainly attack Google's profits.