Kagi Small Web

10 hours ago (kagi.com)

I'm a Kagi search/assistant user and advocate but the "small web" product is a frustrating misnomer.

To me the small web is any little website that was created to be interesting rather than to sell me something. That includes stuff like neocities, "shrine" type sites, single purpose sites, fandom portals, web experiments, etc.

Unfortunately Kagi's definition of "small web" is: blog or webcomic. You must have an RSS feed and it must have recent posts. That rules out so much interesting stuff I don't understand the point.

  • Same feeling here

    Heavy Kagi user and the idea behind small web was appealing; but how its implemented don't click with me

    Their rules excludes an absolute gem like https://www.sheldonbrown.com/ which is, to me, the essence of what we could call the "small web".

    Each times the topic pops up, I try a few random ones and never found anything interesting.

    • This website is the small web - self contained. It's a really good example of the Internet we had and apparently some still want. I think of it like computer graphics where you're definition of space can get bigger as you add a bunch of resources each with their own model space into the relative context of world space. The small web should define how we do that and discover things, not what or how we build within each specific model space.

    • Well, thanks. That small web just taught me in a very concise way a thing or two about bicicle braking technique!

  • Not only that, I just clicked "Next Post" more than a hundred times, and over 90% of posts I got were about LLMs and coding agents.

    • This is a fairly recent phenomenon: I'm a longtime Small Web user and even I struggle with this massive influx of AI posts. I'm hopeful it will be addressed.

  • I am looking for something that would filter for sites that rarely post but have good content. The number one problem with most of these systems is that everything favours frequent posting. Even if I do it manually, I cannot keep the tabs over many rarely posting sites - this is an obvious example of a problem that we delegate to computers. Favouring frequent posters creates incentives to do that even if quality worsens.

    • I'd be fascinated on the economics of this from Google's perspective: specifically the unit economics on generating updated-once-a-year results to queried-once-in-a-million searches.

      Tl;dr: I feel like the long-tail web (90s) was better, but economics pushed high-update-frequency more-centralized results.

  • I could definitely see value in filters for "has RSS" and "has recent posts"—maybe even as the default view—but I absolutely agree that this is much less interesting to me without the wider world of interesting, small sites.

On a similar note, I maintain and grow a manually curated collection of personal blogs with valid RSS feeds: https://minifeed.net/blogs

The criteria is simple: human-written (as much as I can validate myself), in English (for now), with valid RSS feed, and not a micro-blog (so, more than just feed of links or short tweet-like messages).

Similar to Kagi's Small Web viewer, or StumbleUpon-style viewer: you can get a random listing of blogs [1] or a random listing of posts from all blogs [2]. Feeds and posts are indexed, so full-text search works across all blogs. When possible and permitted by robots.txt, text is scraped for searching, so even if some text is omitted in the RSS feed by the author, search should work.

Though I do plan to implement a similar "view one random post at source" kind of view, soon.

UPD: Feel free to submit a blog, including your own! [3]

[1] https://minifeed.net/blogs/by/random

[2] https://minifeed.net/global/random

[3] https://minifeed.net/suggest

  • I wish someone curated a list of sites that are llm written - but are not spam. Just to compare :)

  • The implicit criteria (tech/business and adjacent) is an issue with all these lists for me. But it's also a personal list, which is great. I just wish literally anyone in these had a personal interest on anything else reflected in their lists because I keep checking them and being disappointed.

    A topic that's come up before on here with others doing the complaining about a list I liked for this reason but wasn't top-loaded with tech: https://news.ycombinator.com/item?id=47015676

I've been using the Kagi search engine for months now and I'm not impressed. I bought into it because there were a lot of posts saying that it was "just like old Google" but this has not been my experience. It's the same as new Google, you can type in what you're looking for exactly and you'll get random sort-of related websites.

I remember when you could half-remember a comment from a website, type that into Google, and get taken to the article you were looking for. That was back in like 2010. To me that's the old, and useful, search engine that I want.

  • I switched about a year ago. At the time it did seem like a step up from Google results. But there's been an increasing prevalence of low quality results. Blogspam, AI websites, etc. Obviously not blaming Kagi here, web search has gotten hard recently.

    Is Kagi still better than Google? Probably, I don't really know because I don't use Google anymore. But at this point I feel like I'm with them out of inertia more than being an avid supporter. One of these days I'll re-evaluate Google and decide whether to switch back or not.

    It does occasionally surface interesting results from small sites that you wouldn't get on Google. I do find that to be useful.

    Kagi definitely isn't a bad search engine by any means. Honestly if you haven't used it, try the 100 search free trial on one device. Maybe you'll like it. This feels more like a general decline of the open web.

    • I'm glad to see this comment and the parent comment voted so near the top. I've had the same experience. In my experience, Kagi used to be great... then it became good... and now it's "better than Google".

      "Better than Google" and the fact that I can choose websites to exclude from my search results are two features that I remain willing to pay for, however.

      7 replies →

    • I've been using it for 2.5 years at this point, and have the same experience. I don't think it's hopeless, but Kagi will need to step up their methods. IMO, there's actually a lot they can do here.

    • Everyone has to answer for themselves why they would be OK with Google hoovering up their data in order to deliver substandard results, vs Kagi actively working to remove low-quality results all while collecting no personal data.

      1 reply →

    • It’s definitely not Kagi’s fault. The AI slop is simply taking effect and I feel sorry for them. I never expected them to match Google’s quality, but I was impressed with how close it was when I used it a few years ago.

      1 reply →

  • I've been using Kagi for ~18months and your description doesn't match my experience at all.

    Querying for something like "snowflake json from variant?" in both engines and in google I get a sort-of-right-but-not-really-that-helpful ai summary about "parse_json" function. In Kagi I get an actually useful summary with code examples of parse_json, but also the colon-based syntax for accessing values inside nested objects without needing to parse anything.

    I very rarely need to go into a page, I use Kagi quick search summary with the "?" suffix and it almost always gives me a useful answer in one-shot.

    • First of all, the parent comment's point is that Kagi is often be praised for being like so-called-old-Google[0]. So it's only reasonable to assume they only care about the links, not the LLM summary. What you described is even further from old Google.

      Second, if you want this kind of LLM-digested search result, Google AI studio blows everything out of water (including Google search, obviously).

      [0] I've never bought into the idea that old Google was so much better. But it seems to be a very popular opinion on HN. ymmv.

    • Try g.ai. It's stupid fast and uses google indexes. Kagi? sometimes doesn't correctly parse intent, in Google thing you can just ask function doing this and gives you it, with examples, grounding and extremely fast. I'm paying for kagi since the begging and I guess id cancel it because it gives not so much added value

  • Off topic, but ref

    "I remember when you could half-remember a comment from a website, type that into Google, and get taken to the article you were looking for"

    It's funny to me that (to my knowledge) no browser (mainstream?) implement this functionality yet. Seems like a no brainer to index what the user have actually seen... (Could even be restricted based on viewport - I don't think it's that crazy of an idea)

    I know there's a a number of third party programs which does though. Of course - multi-device being the norm - complicates things.

    • >It's funny to me that (to my knowledge) no browser (mainstream?) implement this functionality yet. Seems like a no brainer to index what the user have actually seen...

      The answer to this is complicated.

      Both Google Chrome and Microsoft Edge actually implement this. Behind the scenes, both will upload your browser history to the cloud. You can see it in network packet captures. It's implemented in the browser for the vendor, but not for the user.

      The choice to not implement this for the user is very deliberate. It's contrary to the vendor's interests if the browser provides this capability directly to users. If a user's browser can take you to a website directly, then you are not using the vendor's search engine, meaning you are not looking at their ads, paid search results, algorithm, etc. It would severly impact their business model.

      This is also the reason why browsers have:

      - Adopted Google Chrome's "Omnibar" instead of a separate address bar and search bar.

      - Implement only basic hierarchical organization for browser Favorites.

      Directly and indirectly, Google is the central nexus of all modern browsers. Aside from Google Chrome, they also:

      - Fund the vast majority of Firefox.

      - Pay Apple for preferential treatment.

      - Provide the same mechanisms to vendors who base their browsers on Chromium (i.e., Microsoft Edge, Brave).

      I would love for this to not be the case. There is hope to be found in small independent browser and search companies/projects.

      9 replies →

    • There are things like Mymind (SaaS) or Karakeep (selfhosted) that do this, though they require you to explicitly save the pages instead of indexing everything by default

      2 replies →

    • If only some operating system incorporated a way to make everything you've seen on your computer locally searchable, wouldn't that be a neat feature?

      3 replies →

    • > no brainer to index what the user have actually seen... I know there's a a number of third party programs which does though

      Which company would you trust with this kind of deep surveillance information on you though?

      1 reply →

  • That's totally fair, though I personally don't share your experience. It could be that we just use search for slightly different reasons.

    One of the reasons I love Kagi is that it respects double-quotes for exact matches. This might seem trivial except I remember being frustrated with both Google and DDG years ago for throwing irrelevant results at me even when I'm querying for an exact match. When Kagi was in beta and I got invited as an early adopter, my feedback to them was that I want a search engine that won't throw crap at me when I'm looking for an exact string match. They've honored that feedback! Even though Kagi doesn't necessarily have the most results, I want double-quotes and things like intitle to actually work as expected.

    Another awesome thing about Kagi is how it lets you prioritize certain domain names. Likewise, it's great for blocking domains completely. All of this has made my search results very clean.

    To each their own. I'm not saying you're wrong, but to me there's no comparison between Kagi's results and every alternative I've tried.

    Oh, another thing I like about Kagi is that it's less censored than Google, Bing, and DDG these days. I used to be a fan of DDG until I noticed that results were sparse or nonexistent for anything even remotely controversial I queried. It became too PG-rated.

  • I’ve been using kagi for about eight months now as well and at least in Europe it’s a significantly better search engine than Google by a long shot. The results are significantly more accurate. I don’t get listicles I don’t get AI spam. I get what I’m searching for, it’s refreshing.

    The assistant is a nice addition but it’s search is superior for me.

  • The only thing that seems to have gotten a lot worse is the trend of ai articles- which isn't kagi's fault but it would be nice if they could figure out how to filter them. They all follow the same patter- "specific thing you want" with a table of contents with loads of repeated chapters and unrelated information, spattered with effectively random images.

    • They’re starting to with their stopslop. Sites that are mostly ai content get flagged and deranked. Still not perfect and I think they only just started working on the backlog of reports so hopefully it holds up helping.

  • I've been using it for over 2 years now. I'm quite happy with it. I like that I don't see adds and my searches aren't being used to target ads against me.

  • I really loved Kagi and was a paid customer for close to two years. But sadly this year I wont be renewing my plan.

    Kagi made search feel just “right” it was simple, got the job done and had some really simple but cool search features.

    But over time they started doing way too much, and I kept seeing more and more features that I really didn't want. It felt like I was paying for all this while I just wanted to type something on to a text box and click search and see a bunch of results organized according to my filters.

    I wish they would just dump all the other nonsense projects like ai and just focus on search only. Or give me an option to pay for search only without any limits.

    • I do agree actually but I’m sticking with them. Their mission of ending slop but also pushing ai tools seem at odds. On one hand they’re marketing to the anti ai crowd while also joining the ai hype? It’s weird.

      2 replies →

  • this is hard to evaluate, but we cannot replicate the old web search experience not just of Google, but Altavista, Lycos or Yahoo, when most of the web is siloed and increasingly botted - simply because the stuff you see in the siloed internet is actively "protected" out of your control

    perhaps the best we can do is this "small web" thing which can be seen as some sort of retrofuturistic solution, but of course the siloed internet is a black hole of content and effort, and of course if the small web gets enough traction, astroturfed generative AI content will target it

  • I’ve been on Kagi for over a year and I’m pretty happy with it. At the beginning there were some noticeable differences in results that frustrated me, but at this point I don’t really miss Google except for some of the nice “not web site results” features like calculation and conversion. I mostly go straight to Wolfram Alpha for those now. And for a lot of the “random curiosity satisfaction” stuff where I would have preferred Google results, I’ll now just use ChatGPT or Gemini.

  • I've switched as of a few years back and it definitely works like pre-AI/search index degradation for me. But I def understand search is very user specific based on how you search and what you are targeting.

  • I feel like I can still do this with Google if I use quotes.

    Kagi I've been using and it's fine. Better than DDG for sure. But sometimes I still go back to google to find something kagi is struggling with.

  • For needle-in-the-haystack searches, I find longer quotes works really well (in Kagi or google)

    Kagi value proposition for me is not the $5 search but the $10 search plus whatever AI chat model you want (I originally did ultimate when I used it for coding). Controllable search and chat satisfies all my one-shot needs.

    I can't really blame Kagi for the web getting bad or for the weak market for secondary search. Part of me wonders if they could use the AI search tools now on the market (now getting lots of investment) instead of the human indexes (subject to monopoly control).

  • throw a question mark on the end to invoke the AI summary results and I find you can get the thing you're looking for as a reference right away. I've used this to dig up forum posts that are over a decade old multiple times with success. Asking the Kagi Assistant for a list of possible links works pretty well too.

    Also on Kagi if you see bad results, you can flag the website to ignore it.

  • I think it's completely unreasonable to assume that anyone would beat Google at the search game, by outgoogling them.

    The reason, that Google is not like it was back in the day is that they are fighting a massive, antagonistic industry designed to game Google. The reason that chatGPT et al improves on search is that there's a effective but very expensive compute layer on top, not that they are better at the Google game. (This extra layer works out fine, because our time is more valuable and Google always came at an insane discount, also thanks to ads)

    • The good news is that Google search results have degraded so much that competitors like Kagi can compete directly. I moved off Google search completely on all devices ~1 year ago and I don't miss it at all, most of the time I forget I have a kagi subscription.

  • Disagree, I love it, at least as good as Google

    • I think that's the problem. I used to find it far superior to google. Now, there are a lot of queries where I am unimpressed with the results and end up trying google just to get better results. (like I used to do with DDG)

      I've had a few experiences now where someone is standing over my shoulder asking me to look something up, and I search kagi, find nothing, then search google and find what they asked me to look up. Then when they ask "what was that other search engine you used first?" I don't feel compelled to vouch for kagi :(.

  • I won't add links so it doesn't look like I'm spamming or promoting a service (though I am, but it seems in line with what you're talking about), but there's a product I've built with my wife which has made things a little bit better in our experience because it gives you an option to choose different providers/indexes, thus tailor results to your personal preference. You can find it from my personal website (my username . com).

  • It's hard to judge one's personal experience with "personalized" search engines. I have personalized search turned off for Google so Kagi is a much better experience for me. I'd recommend leaning more into their feature to lower/block sites from your results, which with Google would require an extension for a similar but degraded experience.

  • > I remember when you could half-remember a comment from a website, type that into Google, and get taken to the article you were looking for.

    Is that even possible today considering there is so much more information and pages around today than in 2010? Old google worked with old Internet. The old Internet does not exist.

  • I typed in my dentist's full business name and location, "<name> family dentistry <city> <state>", and it was still #5 in the results. I still, out of habit, tapped the first link and called that number instead. It's ludicrous. In 2010 that would have been the top hit, next to the Wikipedia page on dentistry.

  • For over two years I’ve maintained the practice of using Kagi and falling back to Google if I couldn’t find something. I can count the number of successes doing that on one hand. In the meantime I get to support a company which actually respects me as a user and isn’t doing things like tying accounts to browsers, AMP (trying to take over the web), trying to kill adblock, etc.

  • I found kagi lacking and very limited unless i paid. Even with a paid sub it didnt feel like good value.

    Im using qwant now and i feel its better.

  • it probably doesn't help that they're constantly bifurcating their tiny team into new projects. their browser is essentially nonfunctional for daily use but they've already moved on to porting it to Linux

  • In comparisons (often shared here) among SERPs, kagi has tended to have fewer blatant results campers crowding out original authoritative sources.

    And yes, Google's founders were right that web ads would kill that experience you want.

  • Kagi uses Google (and other) indexes.

    The main usecase for Kagi is the fact that you can personally uprank/downrank/pin/block sites. And it has a bunch of creature comforts built in like:

    - Attempting to detect AI slop, concatenating listicles ("10 best ...") under one search result heade

    - Attempting to block translated Reddit results

    - Custom lenses that search only coding resources or recipes or whatnot

    - Redirects (so x.com > xcancel.com), although I feel this should be a browser feature

    - Better translate than Google

    There's probably a few things I'm forgetting.

    Kagi is abysmal at image search though. Just assume you will have to use Google for that.

  • I have had a great experience. I can find what I'm looking for and I can block or down-rank sites that are constantly shite. I did find that Google over the past few years has sucked but my Google results were always miles better than most peoples until a couple years ago.

    It's interesting to hear that you can't find what you wanted easily on Kagi.

  • Is that possible today? I have no data but I assume the scale of “the web” grew a couple of orders of magnitude compared to 2012.

  • I am very impressed. Kagi manages to maintain Google-par quality or better most of the time, whereas DDG became an unusable slop pit a few years ago. I'm a very happy customer and happy to keep paying for both Kagi and Orion, in part on principle and in part because the product actually works very well for me.

    I don't even use the AI assistant much, only when there are a lot of disjointed search results and I want a quick summary.

  • Yep that was my experience to. It wasn’t bad necessarily, but certainly not as reliable / dependable as google, and not worth paying for.

    Could just be that I’m familiar enough with google to always be able to make it work for me, could be a frog in boiling water type situation, but… as much as Kagi gets talked up on HN, I was pretty disappointed when I tried it. I was ready to get blown away, and instead I was underwhelmed.

StumbleUpon is that you?

Jokes aside, it's really nice and I can totally see becoming addictive. Kudos to Kagi team for an other user oriented product. (as a side note, I am using Kagi daily and i didn't know about this tool)

  • Yes, SU was fascinating at the time. I kind of like this style of exploring the web; it gets a bit addictive, you spend hours on it but end up finding interesting content and other stuff that you wouldn't otherwise.

  • Im surprised nobody has mentioned cloudhiker which is made to be like stumbleupon.

The first random page it returned to me was this — https://gaultier.github.io/blog/how_to_make_your_own_static_... — which was about building one's own static site generator, which I really liked. I did not realise when I closed that page how hard it would be to find it again, because, of course every new visit to Kagi returns a different page :-)

  • yeah, same happened to me, the first site I was sent to was a list of people sending in random "sunday thoughts" (or whatever it was called) on (actual physical) postcards which then got scanned and posted. There were some good things in there. Now I can't find that site again because I didn't realize it was randomized...

I do love the concept, but a little part of me died each time I came across an article with a very strong AI voice. That just feels antithetical to the ‘small web’ ethos because it obscures the ‘neighbor’ behind it.

I like the idea, but would like to be able to select a language and see the small web of that language. There are more languages than English, and this tool could make them thrive.

Also somehow if they are clever, they could use this for those translation system they are using, but please let us select our own language without feeding automatic translation like youtube does).

  • I think the problem is that it's hard to curate feeds in a language you don't understand. I've been building an uncurated index of OPML blogrolls, with no language restriction. The OPML blogrolls are curated by their owners, so someone decided they met some inclusion criteria, but the overall list is uncurated.

    https://alexsci.com/rss-blogroll-network/

Does it work for you guys to go to about and then click on the "list" link?

For me it says I'm blocked due to hitting a "secondary" rate limit (don't understand what that means). I don't think I've opened a page on github yet today so clearly it's a lie. Is it the referer that triggers this?

In general, freeloading the "small web" on a Microsoft service is kind of ironic. Being blocked by algorithms that try to detect if you're really human is precisely one of the things one would hope to get away from by using small, personal websites

  • If you’re not signed into GitHub, it always throws those errors page (maybe due to scrapping being done with residential IP addresses).

    • Cool, so much for the small web

      No scrapers running on my IP address btw, at least not since it was assigned to me ~10 hours ago (I'm in one of those countries where ISPs seem to have agreed amongst each other that IP addresses must change daily so you can't reliably host things)

Could've at least checked if the website even allows embedding before embedding it, I found two by randomly clicking around that don't.

  • Yeah, many links in the embedded blog posts don't work either, presumably because the target website doesn't allow embedding. On mobile I always have to open them in a new tab for them to work.

StumbleUpon?

My blog was getting traffic from that domain! So that is what that is.

  • Same. I got an influx of bots from Singapore (around 50 visits per day) and in figuring out what's up with traffic, I noticed kagi as a reference for the first time.

    Weird times. People are training their LLMs on my content yet people are still interested in technical content written by a human being. So I guess you just keep writing, right? I find it disheartening to know I'm training LLMs but I think I'm more encouraged knowing there are still humans reading it.

Ah, this might explain the traffic from Kagi a week or so ago. I've been scratching my head over that one. I just checked, and my wee little blog is listed in smallweb.txt. Neat!

Curious what goes on behind the Next Post and Show Similar buttons.

First impressions: My first five pages were stallman.org, a paywalled cybersec newsletter, a German-language blog, an AI-generated blog post ad for a cattle fencing service, and a blog republishing a Disney Parks press release

Bit bummed. The first random page I landed on was a really interesting article for me. The custom cursor (well why not) had me struggling to following a link, and instinctively I refreshed the page. I ended up somewhere else in the haystack with ostensibly no way back to that particular article.

Perhaps I'm yelling into the void here, but what would be great is when first landing at kagi.com/smallweb, the url query parameter would be somehow set, as it is when "Next Post" is clicked.

  • doesn't solve the root problem, but maybe try searching for the topic in kagi with the small web lens?

    • I think it would, so long as the redirected URL with the search parameter was diarized into browser history. It would however introduce a behavior change that may be undesired (users need to know to press "Next Post" instead of refreshing).

      In any case, my Kagi search for the article containing the memorable phrase "rare as rocking-horse s*t" came up empty. Perhaps it's not yet been indexed.

How do we keep getting surprised by enshittification!?

The worst case scenario is that AI runs everything, we have no skills, and are completely dependent on it...and it shows us crummy commercials and subtly steers us to paid placement with no recourse whatsoever. I hate this possible future, but this is where the money will lead.

  • As Kevin Hassett, director of the National Economic Council, said today:

    > "It would hurt consumers, and we'd have to think about what we'd have to do about that, but that's really the last of our concerns right now."

I think it shows the limits of hand curation. It's a tiny, human-reviewed slice of the "small web", only allowing a subset of blogs... but if you select the "programming" category and click around for a short while, you get a fair amount of obvious AI slop.

I don't think it's Kagi's fault, but I guess it's depressing in a way. A lot of "small web" bloggers dream of being a part of the "big web", and when they get a cheat button, they have no second thoughts about mashing it.

Interesting, really like the idea. Maybe in the future a possibility to use it in multiple languages

I run a Hugo blog and I get more interesting referral traffic from Kagi's small web index than from Google at this point. 5,000 curated sites is small enough to be useful most "indie web" directories are graveyards unfortunately..

So, basically, a random site from their index of ~30,000 sites.

You can choose similar sites by index.

But what are the criterion to have your site listed here, or how it will prevent this from just becoming a massive gamified advertising index, or anything more about "why these?" is not obvious to me.

Can anyone explain what is special about these sites specifically, or where this project is going?

A bit off topic, but I noticed I hardly ever use search anymore. It's just google.com/ai in 99% of cases. I believe in the future, search engines must go in this direction ..

Can we just agree that the internet is broken and no amount of boutique search solutions will save it? Kagi, DDG, Google they are all trying to do a search in a pile of steaming sh*, in a hope of finding that shining diamond.

Quite possible that people will come up with a solution eventually. Like Samizdat was a solution to censorship and a broken publishing system in USSR.