← Back to context

Comment by JK-Swizzle

10 days ago

It looks good, but one of the big selling points to me with Kagi is them having their own index. It seems like you guys are working on that, but right now it sounds more like a search aggregator rather than a search engine.

You also mention not using the source for commercial use or distribution, is that only relevant before it becomes AGPL?

I am also struggling to find how to activate the 2h trial, so have not been able to test it out.

7 comments

JK-Swizzle

Reply
BrunoBernardino  10 days ago

Thank you for your comments!

We have an index, it's just not very big, yet. We had a major setback last month with a bug (for less than 24h, the crawler didn't respect robots.txt) and had to delete it entirely, and have been slowly rebuilding it.

You're correct the commercial use is only not allowed before the AGPL comes into play.

You should be able to click on the "top up" link (top or bottom) and see an option for a captcha ("click to prove you're a human"). If you don't, reach out via email (don't share your account number) and I'll give you a voucher for a couple of days.

  • andai  10 days ago

    Interesting. How much does it cost to build a search index in current year? Also, how do you get around the fact that everyone's blocking bots?

    I've been thinking of making a "small web" indexer so I'm curious about that. I'm seeing even tiny websites being behind CloudFlare, Anubis etc. these days. (And everyone complaining they're getting hammered by mysterious distributed HTTP traffic!)

    • BrunoBernardino  10 days ago

      Most of the cost is storage (database, indexes) and memory (RAM, search), and most of the risk is crawlers (they fail, they're reported as abuse, etc.).

      You can look for Common Crawl or Open Web Index for dataset sizes and how many URLs those include to get a sense of baseline storage costs, and then 2x that for minimum usability.

      It's honestly a bit tough to accept that we got a report of abuse and are still dealing with the aftermath of that after having a single crawler go haywire for a few hours (because we play nice and identify ourselves properly), but these... "mysterious" bots that keep hitting all the servers everywhere thousands of times per day just go on like nothing's happening and "no one"'s to blame.

  • JK-Swizzle  10 days ago

    That’s on me, didn’t see the text explaining it above the captcha.

    I’ll give it a shot. Always happy to see some competition in the market. And I love the idea of handing out the source after a year.