Comment by tensor
18 hours ago
Kagi is building their own index. There are also other open indexes. Over time these can replace the big corporate indexes. The hard truth is that the big players in search are dead. They are now the yahoo of search, with landing pages full of ads and results that are primarily ads.
See their revenue (number of paid users is not a secret): something around $7M annually? It was half of that not so long ago (glad, that the userbase is growing).
With their current pricing they are out of their league of having any full-blown index, crawlers, people, what have you.
I would say year ago I was amazed how they are alive at all (unless I am missing something in their funding).
A little over 15 years ago you could index the web with a small cluster. I remember people doing doing it with Cassandra or Elasticsearch. I'm sure you'd need a much bigger cluster, but outside video and images I imagine it's still doable even for a small organization, especially if you're filtering out content farms. Plus, there are many organizations interested in having access to an index, and I'm pretty more than a few currently running their own index and selling to analytics firms.
Index is one thing, great search over it is another.
A competitive, general-purpose web search engine with its own full index is _brutally_ hard and expensive.
This is the reason there are only a few world-class like russian yandex, chinese baidu (to not state the obvious names like google).
1 reply →