Comment by jeromechoo
14 hours ago
Building an index is easy. Building a fresh index is extremely hard.
Ranking an index is hard. It's not just BM25 or cosine similarity. How do you prioritize certain domains over others? How do you rank homepages that typically have no real content in them for navigational queries?
Changing the behavior of 90% of the non-Chinese internet is unraveling 25 years and billions of dollars spent on ensuring Google is the default and sometimes only option.
Historically, it takes a significant technological counter position or anti-trust breakup for a behemoth like Google to lose its footing. Unfortunately for us, Google is currently competing well in the only true technological threat to their existence to appear in decades.
Good news! Google doesn't know how to rank pages either!
yet ... it works "ok" most of the time.
not to mention that people mostly need wikipedia, the news, navigating the infuriating world of websites of big service providers (gov sites, or try to find anything on Microsoft's dark corner of the web), porn and brainrot
but it's awfully hard to make traction on a business that provides this.