Comment by Havoc
14 hours ago
I have a searxng instances but stopped using it after doing a comparison vs actual search provider APIs.
searxng gets rate limited fast and the results are inconsistent depending on what providers it can reach at any given time / isn't captcha'd out.
Meanwhile there are multiple providers with generous free tiers so no point in feeding LLMs inferior data
Also did a couple of side by side tests asking LLMs directly which feed results it prefer for some sample technical searches. Exa and tavily #1, brave #2, searx last
I ran into the rate limiting very fast. Then i realized i had a SerpAPI key. I checked whether SearXNG supports it and it… doesnt, which is bizarre.
> searxng gets rate limited fast and the results are inconsistent depending on what providers it can reach at any given time / isn't captcha'd out.
Exactly. I really wonder what the people going on about using SearXNG are actually doing, because in practice I've seen just what you see. Even from a residential IP, you might get a handful of Google searches before getting backed off. If you ever have multiple searches going on at once, even Bing and Brave are going to back you off pretty quick. My home IP is apparently permanently banned from using Brave Search through SearXNG.
And when you aren't getting results from Google/Bing/Brave, it gets bad really quick for your agent. As much as I wanted to go fully self-hosted for agent search, it just doesn't work well.
Yeah, I had the same issues. I just ended up paying for Brave and Exa. It's literally like, at my volumes, a couple bucks here and there. Nothing to really think about at all. I am a heavy user of AI, but it self-hosted AI, but still the volumes have nothing to worry about. But I'm not trying to scrape Web or anything like that.