Comment by nikvdp
17 hours ago
What are people actually using these hosted search providers for? For personal use I've found using searxng-ai-kit [1] (disclaimer: I'm the author) or similar tools to be far more cost effective (free!). oh-my-pi uses a similar method and will scrape search results for you out of the box, no api key or search provider needed.
Any organization that wants reliable latency, data processing guarantees (it's a big leak vector for company secrets), etc., will use such a search provider over the solutions you outlined.
Especially for chat-like use-cases (in contrast to coding), latency is a lot more critical, especially if you kick off a small single query question. On antother axis, scrape-reliabilty is important for a deep research job that will fire ~10+ web searches and ~100+ web fetches.
e.g. in our chat platform[0], we offer first party MCP servers for web search[1] and web access[2] which work decently well, but some of our customers that prioritize quality over cost, they go with dedicated providers like LinkUp[3]
[0]: https://github.com/EratoLab/erato
[1]: https://github.com/EratoLab/web-search-mcp
[2]: https://github.com/EratoLab/web-access-mcp
[3]: https://www.linkup.so/
Can you explain how this would help protect a company's secrets? I would think that if you want to keep your IP secret surely using something like Deepseek's API directly is not ideal and hosting something locally is best practice.
I personally also wouldn't trust Deepseek with companies secrets. That's why I linked LinkUp, which I know has passed through a quite strict compliance process at a few of our customers.
Obviously, hosting a search index locally would be best practice, for the search part, but usually comes with quite a significant hardware cost and operational complexity, if you want to have an up-to-date index. I'm not aware of good self-hostable options in the space but would be happy to hear about them if anyone here has experience with that! AFAIK LinkUp and some other provides do offer private cloud/on-premise deployments, but I think mostly for their biggest customers.
For the web fetch/access part of the equation, that is inherently live and not containable for a local deployment (unless you would host a whole internet archive/Wayback Machine, which is even more of an undertaking). There, the main options you have to prevent leaks, are a domain allowlist (if feasible), doing query parameter stripping (error prone, and still no 100% guarantee), or configuring the client (e.g. our tool) in a way that users have to explicitly allow those tool calls, and are informed about the posed risks. We are also working on a system, where certain tools are fully deactivated once e.g. an office document that's marked as confidential enters the context.
For the highest requirements on company secrets, an air-gapped system that doesn't have web search/access capabilities is obviously the ultimate solution.
5 replies →
I would use this to discover leaks, not to protect data.
[flagged]
I have a searxng instances but stopped using it after doing a comparison vs actual search provider APIs.
searxng gets rate limited fast and the results are inconsistent depending on what providers it can reach at any given time / isn't captcha'd out.
Meanwhile there are multiple providers with generous free tiers so no point in feeding LLMs inferior data
Also did a couple of side by side tests asking LLMs directly which feed results it prefer for some sample technical searches. Exa and tavily #1, brave #2, searx last
I ran into the rate limiting very fast. Then i realized i had a SerpAPI key. I checked whether SearXNG supports it and it… doesnt, which is bizarre.
> searxng gets rate limited fast and the results are inconsistent depending on what providers it can reach at any given time / isn't captcha'd out.
Exactly. I really wonder what the people going on about using SearXNG are actually doing, because in practice I've seen just what you see. Even from a residential IP, you might get a handful of Google searches before getting backed off. If you ever have multiple searches going on at once, even Bing and Brave are going to back you off pretty quick. My home IP is apparently permanently banned from using Brave Search through SearXNG.
And when you aren't getting results from Google/Bing/Brave, it gets bad really quick for your agent. As much as I wanted to go fully self-hosted for agent search, it just doesn't work well.
Yeah, I had the same issues. I just ended up paying for Brave and Exa. It's literally like, at my volumes, a couple bucks here and there. Nothing to really think about at all. I am a heavy user of AI, but it self-hosted AI, but still the volumes have nothing to worry about. But I'm not trying to scrape Web or anything like that.
I have built my own search engine, which is really great btw. I self-host it.
I’ve combined multiple paid search providers and some that don’t have APIs. These API-less providers I’ve written wrappers for that scrape the normal html pages and return objects from the results.
These wrapped search providers very often block the engine from getting results. Either through rate limits or by blocking my server’s IP.
I am the only user and still get blocked. Let alone if multiple people would hit let’s say DuckDuckGo from the same office IP all day or in a long research session that requires many look ups.
An obvious solution is using a proxy with automatic rotation or some sort of captcha breaking system, which is fine for an agent, but in a search engine scenario might up costing too much in latency.
> I am the only user and still get blocked.
I’m not surprised. Google these days regularly blocks me, a human user with a Google account with decades of tenure.
For OpenWebUI and also OMP with local models. Searxng is, uh, playing fast and loose[1]. More engines are implementing anubis or other challenges, which searxng attempts to implement/bypass, but they are also making these services inactive by default to limit load[2].
Once this cat-and-mouse game goes on a bit longer I expect the quality of free searxng results to crater. My advice to anyone who has any budget at all is to skip over this whole issue and spend a few bucks on a proper search API.
[1] https://github.com/searx/searx/issues/1237 [2] https://github.com/searxng/searxng/pull/6696/changes#diff-e9...
IMO unless you refuse to play the LLM game altogether (then you wouldn’t be having this discussion in the first place), at the end of the day you’re using tech built on top of disrespecting every IP claim and TOS out there on a fundamental level, so no one’s clean and it’s only a matter of how much you can deceive yourself into believing you’re clean.
On a practical level though, using searxng and co. and playing the game of cat and mouse with providers that really don’t want you to use it is just not worth it unless $0.75 per 1000 searches is really expensive to you or you really need the anonymity.
> $0.75 per 1000 searches …[&]… you really [want] the anonymity.
Anybody pooling searches and charging a buck for it with at least a “trust us” privacy promise? Or $1 per 500 even. (Kagi’s more, understandably.) Unsurprising if somebody’s already thought to do this.
1 reply →
I take your point, but there remains a difference between a multi-GB file sitting on my hard drive and my use of a large complicated database fielding my search queries. I don’t believe that anybody is obligated to run a search engine for free for my benefit and it’s logical to pay a fraction of a cent if that makes it work.
How do you deal with fcking cloudflare blocking access to the websites ?
For me, curl-impersonate [0] works great. It makes curl fingerprint identically to Chrome/Firefox. [0] https://github.com/lexiforest/curl-impersonate
I see the Cloudflare challenge even on normal Chrome & Firefox very often, 100% of "first access" (after some timeout) on many sites.
Cloudflare has verified bot program.
Is this what is causing at least part of the massive bot traffic hosts are seeing to their sites? Many direct-to-inference users are just retrieving the site locally every time they need it, let alone sharing site content across users. The big providers like Claude and Google implement their own caching so that when an agent is doing research, if 100k different prompts need the same site content, they don't hit that site 100k times.
Hosted search API from your LLM's provider is much easier to use, faster and less error prone. Also I believe that LLM may work better with its provider's search api, as it was definitely trained on it. I mean, it is just more convenient to use your provider's search.
And if we talking about LLMs, there is no particular point for search to be absolutely free if you still pay for the LLM. It would have more sense if search AND llm would be completely free.
Self-hosting LLMs is much more feasible than self-hosting search, so that seems like a relevant use case.
I ran into a thing running a local dense LLM on DSH (without DeepSeek API key) that doesn't use many MoE experts with vast knowledge. It ends up doing a lot of searching. In the end my local AI is as free as electricity but I pay for the searches.
The search providers I pay likely don't pay their sources. Some others (that I don't use) don't even have the original text only providing summaries. Some offer both.
I am currently using Perplexity for https://veruscite.com/. Verify citations actually exist -- so get in a list of 100, half I can check with crossref, but the other half I need to do a web search and fetch to make sure they exist.
So I need higher rate limits to not make the tool come to a crawl. I probably will not consider DeepSeek API here (I have tried DeepSeek hosted at US providers, like Perplexity). The other search/fetch tools I try do not have great web coverage of Asian journals, but the fact DeepSeek API is not zero data retention I cannot get over.
I've stopped using the browser and have my Pi/LocalLLM(Qwen3.8) do my searching for me. Made a nice interface that will display top results.
No need to pay for a thing, except electricity, which that is the next step, solar powered LLM server.
The only reason this exists today is because normal search providers haven’t been completely disintermediated.
As soon as 100% of traffic to Google searches is not human eyes, they can’t fund it by ad rev, and you will need a service like this
Don't you put yourself at the mercy of bot banning with this approach? It's probably good for poc use cases, but I wouldn't count on it in production.
[flagged]