Comment by thombles
14 hours ago
For OpenWebUI and also OMP with local models. Searxng is, uh, playing fast and loose[1]. More engines are implementing anubis or other challenges, which searxng attempts to implement/bypass, but they are also making these services inactive by default to limit load[2].
Once this cat-and-mouse game goes on a bit longer I expect the quality of free searxng results to crater. My advice to anyone who has any budget at all is to skip over this whole issue and spend a few bucks on a proper search API.
[1] https://github.com/searx/searx/issues/1237 [2] https://github.com/searxng/searxng/pull/6696/changes#diff-e9...
IMO unless you refuse to play the LLM game altogether (then you wouldn’t be having this discussion in the first place), at the end of the day you’re using tech built on top of disrespecting every IP claim and TOS out there on a fundamental level, so no one’s clean and it’s only a matter of how much you can deceive yourself into believing you’re clean.
On a practical level though, using searxng and co. and playing the game of cat and mouse with providers that really don’t want you to use it is just not worth it unless $0.75 per 1000 searches is really expensive to you or you really need the anonymity.
> $0.75 per 1000 searches …[&]… you really [want] the anonymity.
Anybody pooling searches and charging a buck for it with at least a “trust us” privacy promise? Or $1 per 500 even. (Kagi’s more, understandably.) Unsurprising if somebody’s already thought to do this.
CloudFlare is: https://blog.cloudflare.com/introducing-web-search-api
But check the fine print. They talk about the providers guaranteeing ZDR, but I missed the part where they themselves do also.
I take your point, but there remains a difference between a multi-GB file sitting on my hard drive and my use of a large complicated database fielding my search queries. I don’t believe that anybody is obligated to run a search engine for free for my benefit and it’s logical to pay a fraction of a cent if that makes it work.