What are people actually using these hosted search providers for? For personal use I've found using searxng-ai-kit [1] (disclaimer: I'm the author) or similar tools to be far more cost effective (free!). oh-my-pi uses a similar method and will scrape search results for you out of the box, no api key or search provider needed.
Any organization that wants reliable latency, data processing guarantees (it's a big leak vector for company secrets), etc., will use such a search provider over the solutions you outlined.
Especially for chat-like use-cases (in contrast to coding), latency is a lot more critical, especially if you kick off a small single query question. On antother axis, scrape-reliabilty is important for a deep research job that will fire ~10+ web searches and ~100+ web fetches.
e.g. in our chat platform[0], we offer first party MCP servers for web search[1] and web access[2] which work decently well, but some of our customers that prioritize quality over cost, they go with dedicated providers like LinkUp[3]
Can you explain how this would help protect a company's secrets? I would think that if you want to keep your IP secret surely using something like Deepseek's API directly is not ideal and hosting something locally is best practice.
I have a searxng instances but stopped using it after doing a comparison vs actual search provider APIs.
searxng gets rate limited fast and the results are inconsistent depending on what providers it can reach at any given time / isn't captcha'd out.
Meanwhile there are multiple providers with generous free tiers so no point in feeding LLMs inferior data
Also did a couple of side by side tests asking LLMs directly which feed results it prefer for some sample technical searches. Exa and tavily #1, brave #2, searx last
> searxng gets rate limited fast and the results are inconsistent depending on what providers it can reach at any given time / isn't captcha'd out.
Exactly. I really wonder what the people going on about using SearXNG are actually doing, because in practice I've seen just what you see. Even from a residential IP, you might get a handful of Google searches before getting backed off. If you ever have multiple searches going on at once, even Bing and Brave are going to back you off pretty quick. My home IP is apparently permanently banned from using Brave Search through SearXNG.
And when you aren't getting results from Google/Bing/Brave, it gets bad really quick for your agent. As much as I wanted to go fully self-hosted for agent search, it just doesn't work well.
I have built my own search engine, which is really great btw. I self-host it.
I’ve combined multiple paid search providers and some that don’t have APIs. These API-less providers I’ve written wrappers for that scrape the normal html pages and return objects from the results.
These wrapped search providers very often block the engine from getting results. Either through rate limits or by blocking my server’s IP.
I am the only user and still get blocked. Let alone if multiple people would hit let’s say DuckDuckGo from the same office IP all day or in a long research session that requires many look ups.
An obvious solution is using a proxy with automatic rotation or some sort of captcha breaking system, which is fine for an agent, but in a search engine scenario might up costing too much in latency.
For OpenWebUI and also OMP with local models. Searxng is, uh, playing fast and loose[1]. More engines are implementing anubis or other challenges, which searxng attempts to implement/bypass, but they are also making these services inactive by default to limit load[2].
Once this cat-and-mouse game goes on a bit longer I expect the quality of free searxng results to crater. My advice to anyone who has any budget at all is to skip over this whole issue and spend a few bucks on a proper search API.
IMO unless you refuse to play the LLM game altogether (then you wouldn’t be having this discussion in the first place), at the end of the day you’re using tech built on top of disrespecting every IP claim and TOS out there on a fundamental level, so no one’s clean and it’s only a matter of how much you can deceive yourself into believing you’re clean.
On a practical level though, using searxng and co. and playing the game of cat and mouse with providers that really don’t want you to use it is just not worth it unless $0.75 per 1000 searches is really expensive to you or you really need the anonymity.
Is this what is causing at least part of the massive bot traffic hosts are seeing to their sites? Many direct-to-inference users are just retrieving the site locally every time they need it, let alone sharing site content across users. The big providers like Claude and Google implement their own caching so that when an agent is doing research, if 100k different prompts need the same site content, they don't hit that site 100k times.
I ran into a thing running a local dense LLM on DSH (without DeepSeek API key) that doesn't use many MoE experts with vast knowledge. It ends up doing a lot of searching. In the end my local AI is as free as electricity but I pay for the searches.
The search providers I pay likely don't pay their sources. Some others (that I don't use) don't even have the original text only providing summaries. Some offer both.
Hosted search API from your LLM's provider is much easier to use, faster and less error prone. Also I believe that LLM may work better with its provider's search api, as it was definitely trained on it. I mean, it is just more convenient to use your provider's search.
And if we talking about LLMs, there is no particular point for search to be absolutely free if you still pay for the LLM. It would have more sense if search AND llm would be completely free.
I am currently using Perplexity for https://veruscite.com/. Verify citations actually exist -- so get in a list of 100, half I can check with crossref, but the other half I need to do a web search and fetch to make sure they exist.
So I need higher rate limits to not make the tool come to a crawl. I probably will not consider DeepSeek API here (I have tried DeepSeek hosted at US providers, like Perplexity). The other search/fetch tools I try do not have great web coverage of Asian journals, but the fact DeepSeek API is not zero data retention I cannot get over.
Since SEO is big business, and agent driving search is probably the future, I suspect we might see the price of search API's going negative in the future.
Ie. Providers pay you to use their API over competitors, since then they get control of who has influence on the internet. They can probably get sites to pay to be up ranked for specific queries etc
LLM's would be able to answer questions, delineating exactly which facts/assumptions/reasonings are prior work and from where (this URL, that book, etc.), and which parts are it's own deductions, or the user's.
Notable that humans can't cite the source for a huge chunk of their knowledge. Eg. When/where and from who did you learn the meaning of the word "garden".
Your understanding of the world "garden" probably came from 5+ encounters with the word in different contexts, and you cannot separate them - instead your understanding is some kind of average of the knowledge from each encounter.
Interestingly, adjacent and overlapping-meaning words like park, meadow, yard, grassland etc often have disputed boundaries - ie. One person might call something a garden, whilst someone else calls it a meadow. Neither can explain quite why they have chosen their specific word for the object in question, but it might stem from the source of where the meaning of each word was initially learned.
This is the whole Chompsky/Kant thing - the idea of "Poverty of the Stimulus" innate structures. You didn't "learn" a word/concept by direct exposure, but by socialization, association, emotional connections, vibes. LLM seem to learn the hard way.
The Saussure idea of the differential nature of language seems relevant here too: words aren't just lists of things you look up, they are connected to each other and distinguish themselves by being slightly different from the other elements of the clusters they belong to.
Google's goal is not to provide search; it is rather to gather infinite data about people and monetize it. If google offered a search API, it would destroy its surveillance system.
We should have outlawed targeted advertising and Google should have been broken up under antitrust a long time ago.
Ceramic.ai is $0.25/1000 searches. I've been using it as an MCP in OpenCode and it seems adequate. I don't see many people talk about it, and I have not benchmarked it against Exa/Tavilly.
Kagi MCP works really well imo also within DeepSeek Harness. Just looking at cost neglects the possible difference in quality. How good is that DeepSeek web search/fetch?
SerpAPI is comically good at evading Google's scraping protections. If what you want to know is, precisely, what someone searching Google in Pakistan would see, this is the API for you.
is the cost of search and scrape provider is impact the quality too? for the search is yes, because there it shows the quality and depth of the search. but for scrape, wht the cons?
For comparison, Kagi charges $12 per 1000 searches (only talking about their API pricing, like what you’d use with an LLM; their $10/mo plan has unlimited searches but it’s a TOS violation to use those credentials with an LLM).
Thanks for sharing, wasn't aware of this provider. If anyone is wondering about rate limits for the subscription:
>We calculate search rates based on the timestamps of the last searches of a user (just the timestamps, not the queries or anything else!). We keep those timestamps for 12 hours, then discard them. We currently limit searches to 5 per 5 seconds, 15 per minute, and 120 per hour.
In other words, good enough for casual use if you were already going to subscribe. Assuming the rate limits are okay, Uruky’s €5/month becomes cheaper at these search volumes:
How well does this (and similar services) deal with anti-bot protection? What happens if I attempt to retrieve a page protected by the cloudflare checkbox or by Anubis? An error, no useful content, or the actual content I want?
> Search API finds web pages for a query and returns structured results. It is powered by DeepSeek's in-house search engine.
Looks like they've built (or are building) their own index and return cached results. What you do with that information is up to you. If you want to visit the live page, you'd need to find a way around bot detection.
One reason (not the only one!) people use Anubis is to prevent the endless flood of requests from AI companies. If Deepseek scrapes once every few hours, that helps the website owner with the traffic problem while allowing access.
If they don't want to allow access, my guess is that deepseek's search crawler respects robots.txt but I don't know.
I have always blamed Chinese AI operators for the hundreds of thousands of residential proxy requests I receive each day, but there's no way to be sure.
They absolutely dwarf real traffic if left unchecked (I blocked all countries outside my target area).
There is no single answer to this, it entirely depends on the exact service and the exact page
Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections
Meanwhile other services could use a variety of tactics such as simply setting a browser user agent and/or using random data center IPs or residential IPs, etc etc and then the blocking would just come down to whatever the individual page has configured which can actually be pretty hit or miss
From what I've seen, when operating in an environment with bash tools, Claude and ChatGPT will react to a site blocking their normal, attributed fetch tool by calling curl or writing a python script with a different user agent to try and bypass the blocks, rather than respecting the site owners' wishes.
It'd be one thing if they required the user to consciously say "yes, I want you to bypass the block" (still unethical), but the fact that it's just default behavior really says something about the service design alignment.
>Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections
The last part is extremely dubious with some recent reveals.
I've spent a lot of time building datamint.xyz, and dealing with user agents, datacenter IPs, and VPNs gets pretty tricky. What works right now is a fork of the latest Firefox with anti-fingerprinting patches, paired with a high-quality residential proxy network with clean IPs. Even the basic setup needs 11+ code-level patches, and there are still a few edge cases, but this setup should cover 99% of cases.
About a year ago I built my own search and fetch MCP tools for agents based on third-party scraping providers. It is about 15-20x more cost-effective than what OpenAI and Anthropic is offering with their integrated web search and fetch tools. Very impressive that DeepSeek has such a competitive offering today.
ScraperAPI and ScrapingDog. The latter also offers Google Search at a fraction of the official Google API price. Wrap their APIs into an agent/MCP tool et voilà.
DeepSeek Flash really came at a bad time before the Anthropic IPO. I use this thing daily and it’s neck and neck with SOTA models. The search is one of the things setting it apart.
Is anyone actually oneshotting a whole app or even vibecoding one in multiple steps without it being "guided work"? I really wish announcements (like DeepSeek's recent one) would stop focusing on the ability of the model to spit out Minecraft clones on demand.
True, it isn't neck-and-neck. It's neck and balls. Deepseek's balls and Opus 5.5's neck. Whatever you can build with Opus 5.5, I can build in the same time, 1/500th the price with Deepseek Flash 4.1.
If Anthropic or OpenAI fails, the other wins the US market, no? I’m oversimplifying but…that’s a pretty realistic possibility in the face of people thinking sinking one US AI company would hurt the greater US venture into AI.
They're about half a gen behind. Very impressive to get this kind of utility from a flash series model... the impact of RL seems to be behind much of the coding and agentic improvements lately.
Ran a few quick tests on my web-fetch bench. As much as I like DeepSeek, its search and fetch aren't very good (is quite cheap tho). You'd probably be better off using free services like Jina directly, or if you're going paid, Parallel, Firecrawl, Tavily and Exa are all pretty good now.
I wonder if providers do this so they can 'scrape' for 'free' because the customers pay for it? They all seem to have an abstraction around search rather than suggesting the use of native search apis.
I started to think differently about web search for agents, instead of traditional index, I would prefer to have an agentic WALKER that literally walks through pages and finds out whats needed, even if seeded with some initial web search.
I needed to find a word on a wayback archived website. Could not. So I scraped the waybackmachine pages to .md… and found it that way… sorry Archive.org
Most of your local requests will fail (with Python/Curl/whatever) if you are not using a remote-controlled Firefox/Chrome. And even then you will be bitten brutally by captchas, rendering them basically unusable for mass requests. So it's better to outsource your web search to an external service with a stable API.
The existing search machines that offer free APIs for local harnesses are super bad with low quality.
This is worth the invest as LLMs can be guided closer to existing information/code to adapt it.
I am curious if my large website is in their search results. My company charges $1 per page crawled of our website, and the only traffic that worms through our firewall is using hacked residential ISPs. I really need to know who to send the (large!) bill to.
In short, you can try to charge people (or bots) per page but it's not like you'll ever see that money, as it breaks a fundamental paradigm of the internet.
That is a really good idea, in fact I’m doing something similar. I am charging anyone reading this comment 100 bucks per letter read, please tell me who I should send the bill to.
How does that work? I'm not a laywer, but I can invoice you $1 and the T.o.S of reading what I have to say are licensed under a $1 per view licence and by continued reading you consent to the terms of this license. Where can I send the bill for this comment?
Deepseeks browser platform has replaced my typical LLM search routines over the last few months.
Its extraordinary good. They arent trying to sell you things and I cab often find obscure sources/papers.
Its also great for finding research papers in chinese on a given topic.(also its free in the browser and doesn't slow my comptuer to a crawl like most webUIs by llm companies). I have it open all day nowadays.
Edit: How does this get downvoted immediately. Does Anthropic have bots just downvoting any positive sentiment about Chinese AI..
DeepSeek models are amazing and this search pricing is fantastic. All these companies like Parallel, Exa and others are gonna have hard time justifying their hefty valuations. Add Perplexity and many others into the mix. The demand for what they are building is there, but not the margins for the amount of money they raised.
I was really excited to use this (well, I still am), but then I realized Ollama Cloud offer this too, though they don't really mention pricing. I think it's somehow rolled into their subscription.
> These are companies that use hundreds of thousands of stolen credit cards to sign up for fake accounts.
You claim VISA and other credit card companies don't mind if merchants like DeepSeek steal hundreds of thousands of credit cards and use them? And still continue to give them merchant accounts to continue with that?
Isn't it more likely that some people are scaremongering?
Of course credit card companies mind? It's not a problem they can magically solve just because they mind, though. Credit card fraud on massive scale has existed for years and is not going away anytime soon.
> And still continue to give them merchant accounts to continue with that?
The AI companies don't steal the credit cards. They instead fund unaffiliated grey-market transfer stations that steal credit cards. They feign compliance by simply funding another transfer station.
Those companies thought they were interacting only with an entity in a country they trust, and instead had their data sent to a country they don't. If I'm sending you my data I'd like to know beforehand if it's being relayed to another country.
What an absurd question. We have actual evidence of DeepSeek using stolen credit cards and routing conversations to third parties. What evidence do we have of OpenAI or Anthropic doing something similar?
"Anthropic/OpenAI bad!" cynicism is not a replacement for critical thinking (as it increasingly seems to be on HN).
Yes, routing your customers through relays is probably illegal and certainly violates any expectation a business using your API might have.
I don't think that X post from someone that sells relays is meaningful. Obviously Anthropic is aware of what relays are, they aren't dumb enough to conflate relays with DeepSeek.
I think it's important to remember that Deepseek is owned by High Flyer and it's not their core business. They don't need to make a ton of money on Deepseek to stay afloat because most of their income comes from their hedge fund business.
What are people actually using these hosted search providers for? For personal use I've found using searxng-ai-kit [1] (disclaimer: I'm the author) or similar tools to be far more cost effective (free!). oh-my-pi uses a similar method and will scrape search results for you out of the box, no api key or search provider needed.
[1]: https://github.com/nikvdp/searxng-ai-kit
Any organization that wants reliable latency, data processing guarantees (it's a big leak vector for company secrets), etc., will use such a search provider over the solutions you outlined.
Especially for chat-like use-cases (in contrast to coding), latency is a lot more critical, especially if you kick off a small single query question. On antother axis, scrape-reliabilty is important for a deep research job that will fire ~10+ web searches and ~100+ web fetches.
e.g. in our chat platform[0], we offer first party MCP servers for web search[1] and web access[2] which work decently well, but some of our customers that prioritize quality over cost, they go with dedicated providers like LinkUp[3]
[0]: https://github.com/EratoLab/erato
[1]: https://github.com/EratoLab/web-search-mcp
[2]: https://github.com/EratoLab/web-access-mcp
[3]: https://www.linkup.so/
Can you explain how this would help protect a company's secrets? I would think that if you want to keep your IP secret surely using something like Deepseek's API directly is not ideal and hosting something locally is best practice.
6 replies →
I have a searxng instances but stopped using it after doing a comparison vs actual search provider APIs.
searxng gets rate limited fast and the results are inconsistent depending on what providers it can reach at any given time / isn't captcha'd out.
Meanwhile there are multiple providers with generous free tiers so no point in feeding LLMs inferior data
Also did a couple of side by side tests asking LLMs directly which feed results it prefer for some sample technical searches. Exa and tavily #1, brave #2, searx last
> searxng gets rate limited fast and the results are inconsistent depending on what providers it can reach at any given time / isn't captcha'd out.
Exactly. I really wonder what the people going on about using SearXNG are actually doing, because in practice I've seen just what you see. Even from a residential IP, you might get a handful of Google searches before getting backed off. If you ever have multiple searches going on at once, even Bing and Brave are going to back you off pretty quick. My home IP is apparently permanently banned from using Brave Search through SearXNG.
And when you aren't getting results from Google/Bing/Brave, it gets bad really quick for your agent. As much as I wanted to go fully self-hosted for agent search, it just doesn't work well.
1 reply →
I have built my own search engine, which is really great btw. I self-host it.
I’ve combined multiple paid search providers and some that don’t have APIs. These API-less providers I’ve written wrappers for that scrape the normal html pages and return objects from the results.
These wrapped search providers very often block the engine from getting results. Either through rate limits or by blocking my server’s IP.
I am the only user and still get blocked. Let alone if multiple people would hit let’s say DuckDuckGo from the same office IP all day or in a long research session that requires many look ups.
An obvious solution is using a proxy with automatic rotation or some sort of captcha breaking system, which is fine for an agent, but in a search engine scenario might up costing too much in latency.
> I am the only user and still get blocked.
I’m not surprised. Google these days regularly blocks me, a human user with a Google account with decades of tenure.
For OpenWebUI and also OMP with local models. Searxng is, uh, playing fast and loose[1]. More engines are implementing anubis or other challenges, which searxng attempts to implement/bypass, but they are also making these services inactive by default to limit load[2].
Once this cat-and-mouse game goes on a bit longer I expect the quality of free searxng results to crater. My advice to anyone who has any budget at all is to skip over this whole issue and spend a few bucks on a proper search API.
[1] https://github.com/searx/searx/issues/1237 [2] https://github.com/searxng/searxng/pull/6696/changes#diff-e9...
IMO unless you refuse to play the LLM game altogether (then you wouldn’t be having this discussion in the first place), at the end of the day you’re using tech built on top of disrespecting every IP claim and TOS out there on a fundamental level, so no one’s clean and it’s only a matter of how much you can deceive yourself into believing you’re clean.
On a practical level though, using searxng and co. and playing the game of cat and mouse with providers that really don’t want you to use it is just not worth it unless $0.75 per 1000 searches is really expensive to you or you really need the anonymity.
1 reply →
Is this what is causing at least part of the massive bot traffic hosts are seeing to their sites? Many direct-to-inference users are just retrieving the site locally every time they need it, let alone sharing site content across users. The big providers like Claude and Google implement their own caching so that when an agent is doing research, if 100k different prompts need the same site content, they don't hit that site 100k times.
How do you deal with fcking cloudflare blocking access to the websites ?
For me, curl-impersonate [0] works great. It makes curl fingerprint identically to Chrome/Firefox. [0] https://github.com/lexiforest/curl-impersonate
Cloudflare has verified bot program.
I ran into a thing running a local dense LLM on DSH (without DeepSeek API key) that doesn't use many MoE experts with vast knowledge. It ends up doing a lot of searching. In the end my local AI is as free as electricity but I pay for the searches.
The search providers I pay likely don't pay their sources. Some others (that I don't use) don't even have the original text only providing summaries. Some offer both.
I've stopped using the browser and have my Pi/LocalLLM(Qwen3.8) do my searching for me. Made a nice interface that will display top results.
No need to pay for a thing, except electricity, which that is the next step, solar powered LLM server.
Hosted search API from your LLM's provider is much easier to use, faster and less error prone. Also I believe that LLM may work better with its provider's search api, as it was definitely trained on it. I mean, it is just more convenient to use your provider's search.
And if we talking about LLMs, there is no particular point for search to be absolutely free if you still pay for the LLM. It would have more sense if search AND llm would be completely free.
Self-hosting LLMs is much more feasible than self-hosting search, so that seems like a relevant use case.
I am currently using Perplexity for https://veruscite.com/. Verify citations actually exist -- so get in a list of 100, half I can check with crossref, but the other half I need to do a web search and fetch to make sure they exist.
So I need higher rate limits to not make the tool come to a crawl. I probably will not consider DeepSeek API here (I have tried DeepSeek hosted at US providers, like Perplexity). The other search/fetch tools I try do not have great web coverage of Asian journals, but the fact DeepSeek API is not zero data retention I cannot get over.
The only reason this exists today is because normal search providers haven’t been completely disintermediated.
As soon as 100% of traffic to Google searches is not human eyes, they can’t fund it by ad rev, and you will need a service like this
Don't you put yourself at the mercy of bot banning with this approach? It's probably good for poc use cases, but I wouldn't count on it in production.
[flagged]
Since SEO is big business, and agent driving search is probably the future, I suspect we might see the price of search API's going negative in the future.
Ie. Providers pay you to use their API over competitors, since then they get control of who has influence on the internet. They can probably get sites to pay to be up ranked for specific queries etc
@DeepSeek teams, why not train your next series of LLM models with source-aware training?
https://arxiv.org/abs/2404.01019
LLM's would be able to answer questions, delineating exactly which facts/assumptions/reasonings are prior work and from where (this URL, that book, etc.), and which parts are it's own deductions, or the user's.
Notable that humans can't cite the source for a huge chunk of their knowledge. Eg. When/where and from who did you learn the meaning of the word "garden".
Your understanding of the world "garden" probably came from 5+ encounters with the word in different contexts, and you cannot separate them - instead your understanding is some kind of average of the knowledge from each encounter.
Interestingly, adjacent and overlapping-meaning words like park, meadow, yard, grassland etc often have disputed boundaries - ie. One person might call something a garden, whilst someone else calls it a meadow. Neither can explain quite why they have chosen their specific word for the object in question, but it might stem from the source of where the meaning of each word was initially learned.
This is the whole Chompsky/Kant thing - the idea of "Poverty of the Stimulus" innate structures. You didn't "learn" a word/concept by direct exposure, but by socialization, association, emotional connections, vibes. LLM seem to learn the hard way.
The Saussure idea of the differential nature of language seems relevant here too: words aren't just lists of things you look up, they are connected to each other and distinguish themselves by being slightly different from the other elements of the clusters they belong to.
These services only exist because Google refuses to provide a paid search API.
Yes it’s maddening.
It also highlight that Google really isn’t about search, all they care about is advertising and making users profiles.
Have never understood this.
A paid search API seems like it would be easy money for Google. And maybe some lock-in for all sorts of start-ups and enterprises.
There are so many apps that would benefit from search.
I get that it means they can't serve ads.
But you've got to be willing to disrupt yourself or you'll lose eventually anyway.
Better to figure out how to make money from ads and an API, than to cower in fear.
That same fear may be why they failed to be first movers in LLMs.
Have to be willing to disrupt yourself.
At the same time, I've never built a billion dollar company. So I'm absolutely armchair quarterbacking here.
Google's goal is not to provide search; it is rather to gather infinite data about people and monetize it. If google offered a search API, it would destroy its surveillance system.
We should have outlawed targeted advertising and Google should have been broken up under antitrust a long time ago.
> I get that it means they can't serve ads
Easy enough to run the ad-engine over the query, estimate the ad revenue, and bill the user the value of the lost revenue.
They are free to recoup that by serving their own ads or even using Google's ad services.
1 reply →
Google sold search appliances between 2002-2016, the replaced it with Google Cloud Search.
Google provides Vertex for search, but it’s quite slow and so probably only useful for async workloads.
Thought they'd be allover it. Dropping in cheeky ad results.
Everyone would just programmatically filter out the ads.
[dead]
For comparison that’s $5 with Brave Search API and $12 with Kagi API
Ceramic.ai is $0.25/1000 searches. I've been using it as an MCP in OpenCode and it seems adequate. I don't see many people talk about it, and I have not benchmarked it against Exa/Tavilly.
ceramic is very very bad, the worst search provider of all. it feels like it is doing only lexical matching and the index is very outdated
1 reply →
Kagi MCP works really well imo also within DeepSeek Harness. Just looking at cost neglects the possible difference in quality. How good is that DeepSeek web search/fetch?
$4 with Exa, $1.50 with SERP API
SerpAPI is comically good at evading Google's scraping protections. If what you want to know is, precisely, what someone searching Google in Pakistan would see, this is the API for you.
Mojeek is £ 2 per thousand. They have their own index (not a SERP), but subjectively pretty close to Google (I prefer it to Bing/DDG).
$1 for LinkUp fetch, $5 for LinkUp search.
is the cost of search and scrape provider is impact the quality too? for the search is yes, because there it shows the quality and depth of the search. but for scrape, wht the cons?
For comparison, Kagi charges $12 per 1000 searches (only talking about their API pricing, like what you’d use with an LLM; their $10/mo plan has unlimited searches but it’s a TOS violation to use those credentials with an LLM).
>it’s a TOS violation
Competitor uruky is more relaxed about api and is cheaper. Creator is active on hn
https://news.ycombinator.com/user?id=BrunoBernardino
Thanks for sharing, wasn't aware of this provider. If anyone is wondering about rate limits for the subscription:
>We calculate search rates based on the timestamps of the last searches of a user (just the timestamps, not the queries or anything else!). We keep those timestamps for 12 hours, then discard them. We currently limit searches to 5 per 5 seconds, 15 per minute, and 120 per hour.
Source: https://uruky.com/faq
API: https://uruky.com/docs/api
In other words, good enough for casual use if you were already going to subscribe. Assuming the rate limits are okay, Uruky’s €5/month becomes cheaper at these search volumes:
1. Tavily Basic: 1,704 searches/month
2. Brave Search: 2,126 searches/month
3. Exa Fast / Auto: 2,233 searches/month
4. Exa Instant: 3,907 searches/month
5. DeepSeek Search: 7,503 searches/month
6. Ceramic: 22,507 searches/month
4 replies →
Wow! Thank you so much for sharing! Happy to answer any questions about it.
Kagi has an official MCP server [1], where you set the API key. MCP use is always by an LLM, so this seems to be at odds with their TOS.
[1] https://github.com/kagisearch/kagimcp
I believe the commenter was referring specifically to the paid subscription for Kagi, not the per-query usage-based billing API.
Meant as a comparison given the discrepant pricing points:
I pay the Exa premium for semantic retrieval with far higher SNR, plus the academic index [1].
A field like objective can be up to 4096 characters [2] versus ori_question which is limited to 128 here.
[1] https://exa.ai/blog/publications-search
[2] https://exa.ai/docs/reference/search
Have you seen this https://sciencemadereadable.com/ it has free rate limited api end point too.
Artificial analysis' compilation of search api costs per 1000 queries, for comparison:
https://artificialanalysis.ai/agents/search-api?search-api-c...
How well does this (and similar services) deal with anti-bot protection? What happens if I attempt to retrieve a page protected by the cloudflare checkbox or by Anubis? An error, no useful content, or the actual content I want?
> Search API finds web pages for a query and returns structured results. It is powered by DeepSeek's in-house search engine.
Looks like they've built (or are building) their own index and return cached results. What you do with that information is up to you. If you want to visit the live page, you'd need to find a way around bot detection.
One reason (not the only one!) people use Anubis is to prevent the endless flood of requests from AI companies. If Deepseek scrapes once every few hours, that helps the website owner with the traffic problem while allowing access.
If they don't want to allow access, my guess is that deepseek's search crawler respects robots.txt but I don't know.
I have always blamed Chinese AI operators for the hundreds of thousands of residential proxy requests I receive each day, but there's no way to be sure.
They absolutely dwarf real traffic if left unchecked (I blocked all countries outside my target area).
4 replies →
There is no single answer to this, it entirely depends on the exact service and the exact page
Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections
Meanwhile other services could use a variety of tactics such as simply setting a browser user agent and/or using random data center IPs or residential IPs, etc etc and then the blocking would just come down to whatever the individual page has configured which can actually be pretty hit or miss
From what I've seen, when operating in an environment with bash tools, Claude and ChatGPT will react to a site blocking their normal, attributed fetch tool by calling curl or writing a python script with a different user agent to try and bypass the blocks, rather than respecting the site owners' wishes.
It'd be one thing if they required the user to consciously say "yes, I want you to bypass the block" (still unethical), but the fact that it's just default behavior really says something about the service design alignment.
22 replies →
This is not true, Claude and ChatGPT have sent traffic to sites I maintain without any user-agent, it might be they have stopped doing this.
Google has done various versions of this as well, Grok too.
It could be these bots were in some custom harness or something, but they literally sent requests in very weird ways, that humans won't be doing.
I have ended up putting my sites behind vpns or anubis and similar. This is beyond painful.
>Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections
The last part is extremely dubious with some recent reveals.
8 replies →
I've spent a lot of time building datamint.xyz, and dealing with user agents, datacenter IPs, and VPNs gets pretty tricky. What works right now is a fork of the latest Firefox with anti-fingerprinting patches, paired with a high-quality residential proxy network with clean IPs. Even the basic setup needs 11+ code-level patches, and there are still a few edge cases, but this setup should cover 99% of cases.
15 replies →
usually a combination of something like camoufox and a residential ip address
you can bypass those easily except Google Captcha v3, you need to have your own proxy swarm to make it at scale.
About a year ago I built my own search and fetch MCP tools for agents based on third-party scraping providers. It is about 15-20x more cost-effective than what OpenAI and Anthropic is offering with their integrated web search and fetch tools. Very impressive that DeepSeek has such a competitive offering today.
Source for those scraping providers?
ScraperAPI and ScrapingDog. The latter also offers Google Search at a fraction of the official Google API price. Wrap their APIs into an agent/MCP tool et voilà.
DeepSeek Flash really came at a bad time before the Anthropic IPO. I use this thing daily and it’s neck and neck with SOTA models. The search is one of the things setting it apart.
It's absolutely not SOTA for vibe coding a whole app, but for guided work it is great.
Is anyone actually oneshotting a whole app or even vibecoding one in multiple steps without it being "guided work"? I really wish announcements (like DeepSeek's recent one) would stop focusing on the ability of the model to spit out Minecraft clones on demand.
14 replies →
which model is SOTA for vibe coding a whole app? I always use models+agents in guided setup
the moment there is a model which can code whole app, we can fairly say software engineering is done.
1 reply →
It's really not neck and neck with Opus 5.5.
Cost-adjusted,it might be ahead.
It's key to name the boundary conditions. Unlimited cost is a very HN assumption.
1 reply →
Really depends on your use case.
You are right that there's plenty of use cases where Opus 5.5 is better.
2 replies →
True, it isn't neck-and-neck. It's neck and balls. Deepseek's balls and Opus 5.5's neck. Whatever you can build with Opus 5.5, I can build in the same time, 1/500th the price with Deepseek Flash 4.1.
Or at a really good time, depending on who you ask.
Not everyone wants Anthropic to succeed.
If Anthropic or OpenAI fails, the other wins the US market, no? I’m oversimplifying but…that’s a pretty realistic possibility in the face of people thinking sinking one US AI company would hurt the greater US venture into AI.
12 replies →
i thought open models weren't SOTA. therefore not neck and neck... in your experience, do you think it depends on your type of work?
They're about half a gen behind. Very impressive to get this kind of utility from a flash series model... the impact of RL seems to be behind much of the coding and agentic improvements lately.
In my experience, for coding, they are pretty much SOTA. Maybe not for other sciences and problem solving.
Models are comming every 4-6 months. Anthropic, OpenAI, all of them know this.
4-6 months? Try between 8-18 days for a foundation lab release.
> bad time
Regardless of the performance metrics... why bad time?
Deepseek is more like a pet project with huge state backing, and they don't seem to compete with Anthropic in terms of profitability.
Deepseek is profitable.
Anthropic is most likely not. Despite bilions of state funding and contracts.
It literally is not though... even DeepSeek doesn't claim that. It's just super cost-effective
Ran a few quick tests on my web-fetch bench. As much as I like DeepSeek, its search and fetch aren't very good (is quite cheap tho). You'd probably be better off using free services like Jina directly, or if you're going paid, Parallel, Firecrawl, Tavily and Exa are all pretty good now.
I wonder if providers do this so they can 'scrape' for 'free' because the customers pay for it? They all seem to have an abstraction around search rather than suggesting the use of native search apis.
Does anybody use https://www.anysearch.com/ ?
I "fetched" a website with hyperlinks and it only returned the text, which makes it a bit irrelevant?
I started to think differently about web search for agents, instead of traditional index, I would prefer to have an agentic WALKER that literally walks through pages and finds out whats needed, even if seeded with some initial web search.
I needed to find a word on a wayback archived website. Could not. So I scraped the waybackmachine pages to .md… and found it that way… sorry Archive.org
Wouldn’t searching an index be faster? Unless you’re really digging through a site for some super deeply nested link?
I like the idea, but its a surefire way of bloating the context. Although I guess you'd want a subagent to do the search.
Can someone explain why fetch costs money. I thought it was your locally-running harness doing it. How can they charge you for that?
Most of your local requests will fail (with Python/Curl/whatever) if you are not using a remote-controlled Firefox/Chrome. And even then you will be bitten brutally by captchas, rendering them basically unusable for mass requests. So it's better to outsource your web search to an external service with a stable API.
The existing search machines that offer free APIs for local harnesses are super bad with low quality.
This is worth the invest as LLMs can be guided closer to existing information/code to adapt it.
> I thought it was your locally-running harness doing it.
That is rarely the case (see sibling comment for reasons). Even open source harnesses will usually guide you to connect a hosted provider for that.
How does the quality compare to SerpApi?
This is nice. Does search include their LLM data, or does it do real-time search like Perplexity?
We are building our own in-house search API/MCP too, with anti-bot and CAPTCHA workarounds, and we plan to launch Monday morning.
Is there any way to test this without an API key?
I am curious if my large website is in their search results. My company charges $1 per page crawled of our website, and the only traffic that worms through our firewall is using hacked residential ISPs. I really need to know who to send the (large!) bill to.
Your comment reminds me of this classic post: https://www.reddit.com/r/BestofRedditorUpdates/s/uiBsQd8r7f
In short, you can try to charge people (or bots) per page but it's not like you'll ever see that money, as it breaks a fundamental paradigm of the internet.
That is a really good idea, in fact I’m doing something similar. I am charging anyone reading this comment 100 bucks per letter read, please tell me who I should send the bill to.
How does that work? I'm not a laywer, but I can invoice you $1 and the T.o.S of reading what I have to say are licensed under a $1 per view licence and by continued reading you consent to the terms of this license. Where can I send the bill for this comment?
I understand if you pay is a different story.
I mean ... "how does this work?"
they had this for a while now under the Anthropic API
The anthropic API was serving the deepseek search API?
I'm having trouble finding what this actually is
They're entering the search engine scene since they already have "all of the internet" stored due to the expansive LLM crawling -> training?
It is a service for AI agents, so that they can search the web and fetch pages. It is not a product for humans.
As reference point exa charges 4 bucks
I use to manage Google search appliances. This feels a lot like Google old search API product. Nice to see some new players here.
Is AI search where the mysterious relentless crawler traffic is coming from lately?
Not sure, is it worse than two years ago?
I wonder if this is censored. If not then users in China mainland can use the api to workaround gfw (though there are much better ways)
Think there will be two tiers as usual, one for domestic/mainland and one for international users.
Which ones of the search providers listed in this thread use ethical indexing and not outright DDoS?
Deepseeks browser platform has replaced my typical LLM search routines over the last few months.
Its extraordinary good. They arent trying to sell you things and I cab often find obscure sources/papers.
Its also great for finding research papers in chinese on a given topic.(also its free in the browser and doesn't slow my comptuer to a crawl like most webUIs by llm companies). I have it open all day nowadays.
Edit: How does this get downvoted immediately. Does Anthropic have bots just downvoting any positive sentiment about Chinese AI..
what do you mean by "Deepseeks browser platform"? Google isn't turning up much, and searching DeepSeek's docs for "browser" returns no results.
as expected, the typical "politically sensitive" search queries return 0 results
egress on fetch looks to be huawei cloud in hong kong
fetch tool uses a linux firefox 149 user agent with lang set to china. it runs js. could be using something like camoufox.
The rate-limit given there seems really low. 5 requests per second?
I wouldn’t be able to use that API in any app with any significant number of concurrent users.
Or am I reading it wrong?
It's for agentic use
DeepSeek models are amazing and this search pricing is fantastic. All these companies like Parallel, Exa and others are gonna have hard time justifying their hefty valuations. Add Perplexity and many others into the mix. The demand for what they are building is there, but not the margins for the amount of money they raised.
I was really excited to use this (well, I still am), but then I realized Ollama Cloud offer this too, though they don't really mention pricing. I think it's somehow rolled into their subscription.
[flagged]
[flagged]
[dead]
[flagged]
[flagged]
[flagged]
[dead]
[dead]
[flagged]
> These are companies that use hundreds of thousands of stolen credit cards to sign up for fake accounts.
You claim VISA and other credit card companies don't mind if merchants like DeepSeek steal hundreds of thousands of credit cards and use them? And still continue to give them merchant accounts to continue with that?
Isn't it more likely that some people are scaremongering?
Of course credit card companies mind? It's not a problem they can magically solve just because they mind, though. Credit card fraud on massive scale has existed for years and is not going away anytime soon.
> And still continue to give them merchant accounts to continue with that?
The AI companies don't steal the credit cards. They instead fund unaffiliated grey-market transfer stations that steal credit cards. They feign compliance by simply funding another transfer station.
Because OpenAI and Anthropic are so virtuous and trustworthy?
Those companies thought they were interacting only with an entity in a country they trust, and instead had their data sent to a country they don't. If I'm sending you my data I'd like to know beforehand if it's being relayed to another country.
What an absurd question. We have actual evidence of DeepSeek using stolen credit cards and routing conversations to third parties. What evidence do we have of OpenAI or Anthropic doing something similar?
"Anthropic/OpenAI bad!" cynicism is not a replacement for critical thinking (as it increasingly seems to be on HN).
30 replies →
I don’t like OpenAI or Anthropic’s attitude towards the law but this whataboutism.
Completely fake. Those companies used relay stations, or third party gateways, to access Anthropic models. https://x.com/trydotworks/status/2098300730805284961?s=20
Yes, routing your customers through relays is probably illegal and certainly violates any expectation a business using your API might have.
I don't think that X post from someone that sells relays is meaningful. Obviously Anthropic is aware of what relays are, they aren't dumb enough to conflate relays with DeepSeek.
These prices will skyrocket eventually. AI companies need more money.
RAM prices will also probably stay very high as long as AI companis act in such a parasitic manner.
The deepseek philosophy has always been to make a small profit or not do it at all.
I suspect they have spent months building caching infrastructure and designing intelligent fetch algorithms to get down to this number.
Or... CCP gave them a big pile of money to run this as a front for an intelligence operation. Honestly, who knows.
I think it's important to remember that Deepseek is owned by High Flyer and it's not their core business. They don't need to make a ton of money on Deepseek to stay afloat because most of their income comes from their hedge fund business.
> The deepseek philosophy has always been to make a small profit or not do it at all.
That is a very respectable motto for any company.
Will DeepSeek pay the content creators whom pages were fetched?
Is Google paying? Or why do you ask.
If you are talking about google search they direct people to the original website so the owner can monotize the traffic the way they want.
If you are talking about Gemini ( and other AI providers) no they don't they simply steel content.
I am asking to know if DS is any different because I failed to find this information on their website.
1 reply →
I really wouldn't want search engines to pay me to crawl my websites, why would you? The point of publishing stuff online is for people to access it.
Do you?
I am not a service provider. I am a site visitor that would be monetized through ads or other means
Of course no, and that can disincentivize creating more content. You can copy anything and rehash it for next to nothing