Comment by hannasanarion
2 years ago
I'm not able to reproduce the author's bad results in Kagi, at all. What I'm seeing when searching the same terms is fantastic in comparison. I don't know what went wrong there.
In the Youtube Downloader search, NortonSafeWeb is nowhere to be found. I get a couple of legit downloader websites, and some articles from reputable tech newspapers on how to use them or command line tools.
In the Adblock search, ublock Origin is #3, followed by some blogs about ad blocking ethics debates and the bullshit Google has been pulling recently.
In the wider tires grip search, #3 is a physics blog that dives deep into the topic.
In the transistors search, the first reddit link directly answers the question in very similar wording to the hypothetical correct answer spelled out in the rubric. 4/5 of the reddit results are on the correct topic, followed by two SuperUser questinos also on the correct topic, then some linus tech tips and toms hardware articles, also on the correct topic. No Quora questions.
In the vancouver winter snow search, the first several results are from local news papers talking about the anticipated effects of el nino on snowfall, and then a couple of high-quality blogs and weather sites.
Really wondering how Dan got such bad results.
------
Aside from that, the way that the author expects all the results to return the same kind of thing is just... weird? Like, that's not how search engines are supposed to work. A search that gives you 10 links to fundamentally the same thing is a bad search. Search results should cover a breadth of reasonable guesses for what you should be looking for given a query. If you search for "download firefox", and you scroll past the first 5 download links, then you're probably not actually looking for a download link and a blog post about firefox is not "irrelevant" and shouldn't be points against.
This opinion is even borne out in search engine quality metrics that have been industry-standard for decades, like mean reciprocal rank and distributed cumulative gain. What matters is how far you have to scroll to get to a good result, not what proportion of the first N results are good.
What region? I get similarly bad results with international (and a quick check with region US also didn’t improve things) and uBo at only #5, and ytdl at #12. And I already have github on "raise" and a bunch of domains blocked (not many though)
For the transistor query, it’s a very "googly" way of writing a query, when I saw the results I instantly felt like rewriting it and the first try gave much better results with "Why keep cpu transistors getting smaller?". Caveat that the results look better and more topical, I don’t know what a good answer would be, also why I didn’t evaluate the tires or Vancouver weather (I tried a local search for my cities weather, and while the first result was unreleated, the 2nd was okay)
edit: This whole thread made me finally create a file for documenting bad searches on Kagi. The issue for me is usually that they drop very important search terms from the query and give me unrelated results. But switching to verbatim or "forced terms" also prevents any kind of error correction of the search. This used to be one of my main annoyances with DDG back then, and Kagi did not have that issue during the early days.
I have a new Kagi account with no custom rankings and I see the same terrible results. Basically the same as what he describes. yt-dlp is not found at all, the 2010 link to youtube-dl, and a bunch of spam sites.
Same here, I was curious about Kagis low ranking, and couldn't replicate the search results. Also saw ublock Origin on #3, good results for tires, transitors and snow, etc. I've never used any of the Kagi search result weighing features.
Ctrl+F on the page for "System prompt" doesn't show any hits. Given how important those are for ChatGPT (another thought - was the author testing GPT3.5 or 4?) I'm not sure how much weight to put into the ChatGPT results either.
Not sure how much I can take away from this comparison.
I asked GPT-4 about Youtube Downloader and it rambled on about how downloading videos is against Youtube’s TOS and I should buy YouTube premium which has the download feature.
Getting any useful data from GPT-4 about anything even remotely “illegal” is a waste of time.
With a better prompt, you can get it to list some, but it’s very annoying to do so.
Mistral showed that their medium model is far better (yet not good), and the same prompt as in the article gives only one instead of 3 paragraphs of rambling about copyright, and then lists 3 categories of options with examples for each (not good, because ytdl is not one of those listed).
Funnily enough, both mistral and GPT4 apologize profoundly and almost with the same wording when asked "Why did you not mention the very popular, free and open source "youtube-dl" software?" and then mention how/where to get it and how to use it.
3 replies →
Works fine IMO
https://chat.openai.com/share/7dfd22d2-975c-4e6d-ba4b-c6b99e...
https://chat.openai.com/share/90fae0dc-f8fd-4603-835c-5f3a57...
The author already alludes to the fact that you can probably prompt-engineer around this and indeed, as soon as I added a blurb like "these are my own videos that I own the copyright to" it did suggest a bunch of third-party tools and let me ask it about what third-party tools I could use.
It suggested '4K Video Downloader', 'YTD Video Downloader', 'JDownloader' and 'Clipgrab' at first and when I asked for cli tools it came with 'youtube-dl', 'yt-dlp', and 'ffmpeg'
Those seem pretty reasonable results to me but I'll readily admit I don't know (yet) if 'most users' would ask these follow-up questions.
So it has also become one of the glitterati. That didn't take long.
claude.ai produced pretty reasonable results.
I'll second the chorus of those curious to hear how you've customized the search engine. I was able to reproduce the lackluster results, and was sadly disappointed. I expected what you seem to have found, that Kagi would outperform.
A specific example: for "ad blocker" the first result was some paid ad blocker and ublock was down the page below the fold.
I use Kagi because I'm trying to remove Google from my life, but their text search is worse than Google in my experience, and the image search is abysmal. I'm wondering how long I can keep this up. I already revert to Google for image search, and am finding myself using either Google or ChatGPT over Kagi more and more for text as well.
Kagi had a pretty substantial image search update just few days ago [1]. Do you still the issues with it?
[1] https://kagi.com/changelog#2793
Good info - will experiment!
It's already performing better on a (n=1) test I tried.
"Talos Principle 2". (Video game sequel) Previously (~5 days ago), Google returned various screenshots etc from the game `The Talos Principle 2`. Kagi returned mostly results from `The Talos Principle (1)`. Now the latest Kagi results are a mix, mostly from 2. So, it does look like it fixed this query.
have you customized your results and lowered or raised many domains?
Kagi is awesome for me too. I just realize using Google somewhere else because of the shit results.