My number one question about search APIs is always if they allow you to store and resyndicate results you get from them.
If I'm running an agent system but I'm not allowed to store the responses - or provide a "share transcript" button - that's a pretty significant limitation.
The answer to that question is inevitably buried deep in the terms. Here's the relevant section I found for Ceramic, in their list of things you can't do:
> (n) collect, aggregate, store, or compile Output, including search results, relevance scores, or rankings, for the purpose of creating or contributing to any database, dataset, index, or corpus, whether or not such database, dataset, index, or corpus is used for a purpose that competes with Ceramic; (o) resell, syndicate, or otherwise make Output available to any third party on a standalone basis or as a separately accessible component of another product or service; provided that you may display Output to your authorized end users within your own application so long as such Output is integrated into your application's functionality, is incident to the end user’s real-time query, and is not independently accessible, extractable, or downloadable by end users or third parties; or (p) retain, cache, or store Output beyond what is reasonably necessary to display such Output to your authorized end users in the ordinary and real-time course of use, unless expressly permitted in an applicable Order Form.
It seemed like this part gives you the exception you wanted:
> provided that you may display Output to your authorized end users within your own application so long as such Output is integrated into your application's functionality, is incident to the end user’s real-time query ...
but it continues:
> ... and is not independently accessible, extractable, or downloadable by end users or third parties
How can you prevent end users from extracting it if its visible? Why even have the exception if you just throw it out with an impossible to meet restriction like this?
Not to mention: "retain, cache, or store Output beyond what is reasonably necessary to display such Output to your authorized end users in the ordinary and real-time course of use" which would seem to preclude storing it in a long lived session.
My general stance on things like this is to think about the intent -- why does the company have that in their TOS. Use that as a proxy for assessing the likelihood of the company enforcing the terms against you.
This is the right way to think about it. If there's any fear of getting caught, just use a reputable VPN or one of the hundreds of residential proxy providers.
For those developers out there, the best is still Gemini Flash Lite 2.5 believe it or not. It gives you 1000 google searches per day for free. Compare to Flash Lite 3.x which is 5k PER MONTH and then a few pennies PER SEARCH. Nuts. Didn’t realize search was so expensive.
Perhaps realizing all of this, Google hasn’t yet deprecated 2.5, bit limits access to it to “those who have used it before.”
So I wish I could use Google for https://veruscite.com/, but the number of Google searches are a hard cap on the account! So yes that is fine for agentic coding, but for an app that relies on web-search is not sufficient.
I am currently using Perplexity fast search and fetch, and I am happy with that. I would try our Ceramic.ai, but I need to be able to fetch the pages as well (I do not want summaries).
(I work at Linkup.) We do both: search returns raw results, no summaries, and there's a separate fetch endpoint that returns the full page as markdown, with optional JS rendering. You should compare us against your Perplexity setup - you might some value in switching
Can Gemini Flash Lite 2.5 be made to return raw search results. Some 'search' providers I looked at returned summaries, or vector relevance matches (of presumably a smaller/stale page set).
1. Google News API now returns only Google links that don't resolve to anything in code.
2. Google Search results are atrocious and only unearth non-authoritative blogspam and aggregator sites.
That’s the enterprise agent one, still available for API. Oh and I got it wrong… it’s actually 1500 searches per day free! Gemini Flash 2.5 also has this btw but I prefer Flash Lite to reduce costs. It’s nuts when you compare cost and features to any of the 3.x models. https://ai.google.dev/gemini-api/docs/pricing
I think Cloudflare is (for companies already using it) approaching the status of trusted main cloud supplier (which usually would be AWS, GCP, Azure) via which the majority of cloud costs are billed (so you don't have to go through a fresh procurement process).
Replace trusted with convenient. They're glowing pretty hard giving out all that stuff very cheap in exchange for being the middle man on everything. Not that I mind for my trivial use case.
This practice of having the one provider should be eliminated. Companies self-inflict lock-in to large platform providers, prevent their own teams from using better technology options and stifle innovation. It's crazy that even with a pile of SOC/ISO/PCI/HIPAA/NIS certificates, procurement is still a months-long process, it should be much easier to do business.
I think their strategy is: "AI coding means we can build everything. Our infrastructure approach is incredibly quick to build upon, so why not build it all ourselves and then anyone with half a brain will move their stuff to Cloudflare and leave AWS in the dust."
I've been using the web search in OpenRouter, which is similar in that it's a wrapper around other search engine providers. It's really convenient to be able to experiment with new models and new search engines without having to go through corporate hoops to subscribe to a new service.
To add to this, some organizations just prefer using one provider for their cloud service. So if they build on Azure/Google Cloud/AWS, then everything needs to be on there. Cloudflare probably wants to offer the same here, where everything can be built on Cloudflare.
Not sure where the trust claim lands, but the first two are now exceedingly trivial with code agents. A little more work perhaps, but not hard at all. I’ve done this myself (not with those providers) with several search platforms.
Cloudflare are setting themselves up as the arbiter who will decide which requests are a) human, b) authorized AI bots, c) illicit/banned bots.
Given the number of people on HN who report massive problems from scrapers and other bots, it sounds like if Cloudflare doesn't do this, someone else will need to. I might have thought bandwidth was cheap enough now for it not to matter, but I guess the bots are costing some sites a lot of money.
Cloudflare seems to be very excited to eventually get a 30% cut on pay-to-crawl.
As for the bots, I thought the same thing, but it is indeed a huge problem. They've brought my websites down pretty frequently recently. I tried Cloudflare but visitors complained, and I think you can't win against the bots anyway, so I've resorted to performance improvements and serving every request.
It isn't bandwidth that causes problems. It is the various types of load that they can add to your servers. Which is why no centralized vendor can decide the proper caching or what bots should be blocked, allowed, rate limited, etc. Those things do not have standard answers - it depends on what your apps do, your audience, their usage patterns, and sometimes the regulatory environment in which you run.
Cloudflare doesn't block bots. It's trivial to use residential proxies and your very obvious bot will only get blocked maybe 5% of the time when using rotating IPs.
My coding agent uses the hister cli, i.e. a local index. That often requires me to seed it manually as a downside. The upside is that it caches website contents via browser plugin, which is a nice workaround for bot blocking.
What do you mean by seeding it manually? Like do you programmatically "browse" to bolster your hister index? Asking because I started using hister a month or so ago and have been really liking it, and I'm curious how others are using it.
Yes, I browse around, open a dozen tabs so they get indexed, and close them without reading. Then a coding agent is pretty good in composing a report from that index. Generated these recently: https://qznc.github.io/sloppy_research/en/
Hister tells me my index is currently 39208 pages.
I was just looking into these as DeepSeek Harness w/ Qwen3.8-27B relies heavily on search. I was going to go with Serper.dev[0] $1 per 1000 (or lower in quantity).
The providers[1] behind this Web Search API have very different rates:
Ceramic.ai: $0.25 per 1,000 requests
Linkup: $5.00 per 1,000 requests
Exa: $7.00 per 1,000 requests
Because it's their release week, so there's multiple new products every day. The overlap of people using HN and Cloudflare is pretty large, so not that surprising.
A related Q: why are their products so popular? I get why CDN/DDOS protection is but what about everything else? I have never ever found a use for stuff like Workers. (Sincerely asking, not dismissing them as useless)
Workers is compute-on-demand and particularly only pay what you use. Especially in the current days, something like Workers is infinitely appealing if you don't want to manage or pay for a dedicated server, assuming the service you want to run can be completely hosted (or a replacement vibe-coded) for the Service Workers API / deployed to CF's platform.
You also don't pay for actual data transfer, so the billing is overall simpler - AWS and GCP have similar per-request compute options, but every part of the platform has extra fees (like per-gb data transfer billing, sometimes you need a VPC to interconnect services, secrets being an extra charge, etc).
Workers have a nice and easy deployment model (when it's not broken) compared to AWS lambda, so I get why people are tempted. It's one simple file compared to 4 separate pieces of infra. But yes, please, use anything else that doesn't pay for the CloudFlare protection racket. For example there's https://bunny.net/edge-scripting/
Pages is a very convenient way of deploying static websites/SPAs with a generous free tier. You just need to find the tiny links in their dashboard to avoid accidentally using workers instead (which is supposed to supersede it but is clearly worse for this usecase).
The workers paid tier is $5/month and you can do a LOT with that. Once you drink the cloudflare kool-aid regarding workers they let you build very scalable apps while jumping through many fewer hoops than you would on AWS/GCP, at a fraction of the cost.
Workers gives me a free static site, just dropped the .html file there (or connect to git).
CDN, DDoS protection, excellent DNS hosting features, web monitoring, web analytics, advanced web and service filtering and blocking, zerotrust networking / vpn options, a solid API that works great with terraform, tons of other stuff.
I suspect also to do with internal Slack etc. where employees vote on launch posts (a lot of them on HN since long). Not a scam or accusing anyone but this probably propels a lot.
Many people have outsourced the decision on who can access their websites to CloudFlare ("bot protection"), which incidentally makes these websites harder to access by bots working for humans.
Now there is an official paid search API, and I'm guessing the certified providers will be allowed through the Cloudflare "bot protection"?
Yeah, I keep wondering if I should switch to something cheaper, but I'm too lazy to evaluate service qualities across providers, and I don't really use enough for it to matter. If they are the most expensive because they're the best, I'm fine with that, but have no idea.
Let's not conflate crawlers with the traffic that bot protection services block. A crawler that respects robots.txt is a good internet citizen and can provide a vital service.
However, so-called AI crawlers are not the same as crawlers of yore. They hit live pages every time a user prompt triggers a web search.
This Web Search API, unlike an AI crawler, only fetches periodically. It feels like a step in the right direction for managing resource strain across the internet. If only the LLM giants could do something similar.
Tried one query on ceramic.ai (the default provider for cloudflare web search api): "qwen-3.8 flash next and rtx 5090 best inference setup" ... 0 results ... same query on google and ddg both yield proper results.
Then shortened the query to just "qwen-3.8 flash next" ... results came.. all unrelated. In fact, these were almost all paper links .... no relation to actual search term.
And I had thought that I finally had found a cheaper search alternative.
You know the craziest part? This time I searched for their own website address: "ceramic.ai" .. results came... none pointing to the website or any page on it.
Then searched for "Cloudflare OHTTP Gateway" .. this text is literally in the title ... but zero link for this page.. the closest it yielded was this link: "https://developers.cloudflare.com/privacy-gateway/" ... it seems cloudflare updated this 2 days back.. the original content was last updated in 2022 ... so that's what the cutoff index seems to be.
I made a zero ads SERP using one of these "AI first" search providers: https://github.com/scosman/froogle (live version https://froogle.fyi). In this case Keenable.ai. Generally the same pattern: it's not usable.
Surprised no one in this thread has mentioned running a self hosted search API.
I personally used to use Firecrawl's paid credits (got a bunch of em for free at an event) before I realized that they allow you to self-host your own instance (albeit missing some features I never use anyways).
It's been working really well for my agents, I even hosted a small observability tool that proxies the requests so I can see how many are failing and the percentages are always below 2%.
Spent a few months building a product scraper using a mad mash up of various LLMs, OCR, etc. The pricing for their providers is 3x-8x higher than something like Luna 5.6 w/ Web Search. Not sure what their differentiator is, unless they just wanted to launch something.
You are conflating a couple of different things here
There actually is such a thing as verified bots on Cloudflare that gets through most blocks (and these services are likely are part of that), but ultimately it just depends on how the website owner has things set up in Cloudflare
Anthropic plausibly uses Brave Search... and Brave search maintains its own index. Makes sense: cheaper search API, leveraging non-Google, etc.
Here we are, one layer of indirection more: Ceramic, Exa, Linkup. Who knows what they use. If you told me that those 3 build and maintain their own index, I'd first question whether that was true, and if it is, I would question whether it was any good (relative to Google/Bing/Brave).
So what is CF providing here? Maybe some free credits to entice us to use their router? No, not that either ("billed to your AI Gateway credits"). Maybe a comparison of which agent search yields the best results? Nope.
It's a crappy proxy- probably less efficient and more volatile than hitting the agent API directly.
This is only if I understand the product correctly (which I admittedly skimmed) due to the sheer number of screeching vibey nothingburgers coming out of CF over the past month.
>Here we are, one layer of indirection more: Ceramic, Exa, Linkup. Who knows what they use. If you told me that those 3 build and maintain their own index, I'd first question whether that was true, and if it is, I would question whether it was any good (relative to Google/Bing/Brave).
Hi! Exa Head of Index here. We certainly do have our own index and it's one of the biggest among the independent players (i.e. not Google and Bing, which by the way closed off their official search APIs). [1]
Regarding the quality: search is a multi-dimensional problem, you can be better on one set of queries and worse on the other. There are tons of benchmarks in the industry, all the players in the AI search market are fighting very hard to climb to the top, updates are shared every week.
We track dozens of use cases and run evals continuously, we perform well on all the verticals we optimize for. Not only we top the ranking on e.g. financial queries, but also Claude with Exa search performs better that Claude with native search -- as measured by independent observers [2]. This means that the underlying search is materially better for the outcome, it's not just how we evaluate the search itself.
I am pretty sure exa specifically say it trains on your data in it's privacy policy, so how can it be ZDR ?
I remember as I was looking at the available web tools for hermes agent not to long ago and looked through the keyless web providers privacy policies, which exa is one of them.
Does anyone have insights on the quality differences? Web search API pricing for AI agent usecases has always felt so expensive for what it is, but I have no grounding on the economics of running a web index.
I am trying to understand the value.. This is for customers who have their agents already on CF? Improved latency & same eco-system etc., Right? Because others can always use Google Search APIs
Cloudflare has been shipping more than FedEx lately. Would love to learn more about how they are going about this from a strategy, planning and execution standpoint.
Wow I guess I am in the minority of folks building a search engine for humans now, this is a wild business model but best of luck to the 3 search index providers sitting behind this proxy, I hope it's worth their while financially speaking. Building an index is hard and expensive (I know).
Weird choice by CloudFlare, would been great if they have shared why it was created.
I use CloudFlare developer platform and quite happy with tools, but I didn’t use the gateway API and always used OpenRouter which does support web search.
I can see it useful for those who didn’t do any integrations or like to keep logs at one place, but did customers actually ask for this?
I run a small side project on Cloudflare Workers, so having search available right from a Worker without adding another vendor is appealing. Curious how the pricing compares to Brave's search API.
I guess I'm not understanding the value here - to compete with Google and the likes, the scale, cost and complexity would be huge. Appreciate new entrants in an existing field but not seeing this one.
>"to compete with Google and the likes, the scale, cost and complexity would be huge."
CloudFlare's entire business is scale, cost, and complexity. They are powering like half the web at this point. Wouldn't really call them a "new entrant".
Codex and Claude Code need to do countless web searches, I'd guess they have a partnership with Google. Open models don't have this partnership so the search API needs to come from somewhere.
SearXNG works pretty well for my personal agents. FYI it's a free search gateway you can host locally, and there are many public instances. It's like the old days when many different people provided the same free service for all.
I wish non-fuzzy searching was still trendy. It sucks when I know what I need and remember a bunch of keywords from the page, but search engines return either 0 results or a bunch of results that don't even contain the keywords.
the zero-retention promise from the search providers is useful, but the requests also show up in gateway logs. can you keep the billing data without storing the actual search queries?
Search is an interesting building block for agentic systems. The challenge isn't just retrieving results, but deciding what to search for, evaluating the results, and determining when the information is sufficient to move to the next step.
As AI systems increasingly use search as a tool, the quality and reliability of that tool become an important part of the overall agent workflow.
My number one question about search APIs is always if they allow you to store and resyndicate results you get from them.
If I'm running an agent system but I'm not allowed to store the responses - or provide a "share transcript" button - that's a pretty significant limitation.
The answer to that question is inevitably buried deep in the terms. Here's the relevant section I found for Ceramic, in their list of things you can't do:
> (n) collect, aggregate, store, or compile Output, including search results, relevance scores, or rankings, for the purpose of creating or contributing to any database, dataset, index, or corpus, whether or not such database, dataset, index, or corpus is used for a purpose that competes with Ceramic; (o) resell, syndicate, or otherwise make Output available to any third party on a standalone basis or as a separately accessible component of another product or service; provided that you may display Output to your authorized end users within your own application so long as such Output is integrated into your application's functionality, is incident to the end user’s real-time query, and is not independently accessible, extractable, or downloadable by end users or third parties; or (p) retain, cache, or store Output beyond what is reasonably necessary to display such Output to your authorized end users in the ordinary and real-time course of use, unless expressly permitted in an applicable Order Form.
https://www.ceramic.ai/terms-of-service
Am I alone in caring about this?
It seemed like this part gives you the exception you wanted:
> provided that you may display Output to your authorized end users within your own application so long as such Output is integrated into your application's functionality, is incident to the end user’s real-time query ...
but it continues:
> ... and is not independently accessible, extractable, or downloadable by end users or third parties
How can you prevent end users from extracting it if its visible? Why even have the exception if you just throw it out with an impossible to meet restriction like this?
So they crawled the web, stole the information to populate their own database and then pretend it was "illegal" for others to steal it back?
The weird attitude in the Internet Tech company scene is akin to Gold Rush scenarios.
Who are the native people?
11 replies →
Not to mention: "retain, cache, or store Output beyond what is reasonably necessary to display such Output to your authorized end users in the ordinary and real-time course of use" which would seem to preclude storing it in a long lived session.
1 reply →
My general stance on things like this is to think about the intent -- why does the company have that in their TOS. Use that as a proxy for assessing the likelihood of the company enforcing the terms against you.
This is only valid up to the level of risk you can tolerate for them pulling the rug out from under you.
1 reply →
You are not alone, I agree that this is a frustrating limitation.
It’s a shit tier web scraping startup, just violate their terms, who cares.
This is the right way to think about it. If there's any fear of getting caught, just use a reputable VPN or one of the hundreds of residential proxy providers.
1 reply →
For those developers out there, the best is still Gemini Flash Lite 2.5 believe it or not. It gives you 1000 google searches per day for free. Compare to Flash Lite 3.x which is 5k PER MONTH and then a few pennies PER SEARCH. Nuts. Didn’t realize search was so expensive.
Perhaps realizing all of this, Google hasn’t yet deprecated 2.5, bit limits access to it to “those who have used it before.”
It’s really really good for low cost search!
So I wish I could use Google for https://veruscite.com/, but the number of Google searches are a hard cap on the account! So yes that is fine for agentic coding, but for an app that relies on web-search is not sufficient.
I am currently using Perplexity fast search and fetch, and I am happy with that. I would try our Ceramic.ai, but I need to be able to fetch the pages as well (I do not want summaries).
(I work at Linkup.) We do both: search returns raw results, no summaries, and there's a separate fetch endpoint that returns the full page as markdown, with optional JS rendering. You should compare us against your Perplexity setup - you might some value in switching
1 reply →
Can Gemini Flash Lite 2.5 be made to return raw search results. Some 'search' providers I looked at returned summaries, or vector relevance matches (of presumably a smaller/stale page set).
No raw results unfortunately.
I can't use Google for anything anymore.
1. Google News API now returns only Google links that don't resolve to anything in code. 2. Google Search results are atrocious and only unearth non-authoritative blogspam and aggregator sites.
> Perhaps realizing all of this, Google hasn’t yet deprecated 2.5, bit limits access to it to “those who have used it before.”
Don't give them (G) ideas.
The idea i do want to give them… guys, differentiate your Gemini models with free to low cost search. It’s what your known for! Lean into it.
2 replies →
Is this with the $20/mo Google AI Pro plan?
I believe they're referring to the free tier of Google AI Studio. [1]
[1]: https://aistudio.google.com
"This model is being retired on October 20th, 2026"
Google AI's deprecation page [0] says that there is "No shutdown date announced" for gemini-2.5-flash-lite.
(Was your comment a joke? Or did google announce this through different channels?)
0: https://ai.google.dev/gemini-api/docs/deprecations
2 replies →
That’s the enterprise agent one, still available for API. Oh and I got it wrong… it’s actually 1500 searches per day free! Gemini Flash 2.5 also has this btw but I prefer Flash Lite to reduce costs. It’s nuts when you compare cost and features to any of the 3.x models. https://ai.google.dev/gemini-api/docs/pricing
Also if you use a newer API key (I think mine is from like Feb of this year) then you'll get a "this API key is too new to use 2.5" message.
Can someone clarify how this works? Why does an LLM give me a search API? The search APIs I've always used were just "post query, get JSON".
The LLM does the searching but is still limited
1 reply →
Why not use those providers directly? Does Cloudflare need to be in the middle of everything?
It would be difficult for them to provide intelligence to the US without being in the middle of everything.
[flagged]
I think Cloudflare is (for companies already using it) approaching the status of trusted main cloud supplier (which usually would be AWS, GCP, Azure) via which the majority of cloud costs are billed (so you don't have to go through a fresh procurement process).
I don't know what you mean by "trusted", but how many times do folks have to go through the same loop?
I'm with OP - a company that wants to insert itself in the middle of everybody's business is not being altruistic, they're playing the long game.
4 replies →
Replace trusted with convenient. They're glowing pretty hard giving out all that stuff very cheap in exchange for being the middle man on everything. Not that I mind for my trivial use case.
This practice of having the one provider should be eliminated. Companies self-inflict lock-in to large platform providers, prevent their own teams from using better technology options and stifle innovation. It's crazy that even with a pile of SOC/ISO/PCI/HIPAA/NIS certificates, procurement is still a months-long process, it should be much easier to do business.
1 reply →
I think their strategy is: "AI coding means we can build everything. Our infrastructure approach is incredibly quick to build upon, so why not build it all ourselves and then anyone with half a brain will move their stuff to Cloudflare and leave AWS in the dust."
I've been using the web search in OpenRouter, which is similar in that it's a wrapper around other search engine providers. It's really convenient to be able to experiment with new models and new search engines without having to go through corporate hoops to subscribe to a new service.
Ease of integration and billing. Failover. Higher trust.
To add to this, some organizations just prefer using one provider for their cloud service. So if they build on Azure/Google Cloud/AWS, then everything needs to be on there. Cloudflare probably wants to offer the same here, where everything can be built on Cloudflare.
Not sure where the trust claim lands, but the first two are now exceedingly trivial with code agents. A little more work perhaps, but not hard at all. I’ve done this myself (not with those providers) with several search platforms.
2 replies →
Curious what other people’s experience is with cloudflare billing. When you go through an AE, everything seems made up anyways.
Why would anyone trust Cloudflare?
13 replies →
How else you are going to make them give you search second party API's, they just bridge it for you reliably
Cloudflare are setting themselves up as the arbiter who will decide which requests are a) human, b) authorized AI bots, c) illicit/banned bots.
Given the number of people on HN who report massive problems from scrapers and other bots, it sounds like if Cloudflare doesn't do this, someone else will need to. I might have thought bandwidth was cheap enough now for it not to matter, but I guess the bots are costing some sites a lot of money.
Cloudflare seems to be very excited to eventually get a 30% cut on pay-to-crawl.
As for the bots, I thought the same thing, but it is indeed a huge problem. They've brought my websites down pretty frequently recently. I tried Cloudflare but visitors complained, and I think you can't win against the bots anyway, so I've resorted to performance improvements and serving every request.
It isn't bandwidth that causes problems. It is the various types of load that they can add to your servers. Which is why no centralized vendor can decide the proper caching or what bots should be blocked, allowed, rate limited, etc. Those things do not have standard answers - it depends on what your apps do, your audience, their usage patterns, and sometimes the regulatory environment in which you run.
Cloudflare doesn't block bots. It's trivial to use residential proxies and your very obvious bot will only get blocked maybe 5% of the time when using rotating IPs.
2 replies →
You tell us, you're the president of bonsai.io.
Why would I use bonsai? Why not use ElasticSearch directly?
National security, bro.
My coding agent uses the hister cli, i.e. a local index. That often requires me to seed it manually as a downside. The upside is that it caches website contents via browser plugin, which is a nice workaround for bot blocking.
Thanks asciimoo for https://github.com/asciimoo/hister
> caches website contents via browser plugin
does it? I am running it but was under the impression that it did not cache the content I am viewing, unlike SinglePage.
Why do you have to seed the pages manually?
The original request via the MCP is somehow blocked?
Something on my todo-soon list is to figure out how to import the devdocs.io doc bundles into hister.
What do you mean by seeding it manually? Like do you programmatically "browse" to bolster your hister index? Asking because I started using hister a month or so ago and have been really liking it, and I'm curious how others are using it.
Yes, I browse around, open a dozen tabs so they get indexed, and close them without reading. Then a coding agent is pretty good in composing a report from that index. Generated these recently: https://qznc.github.io/sloppy_research/en/
Hister tells me my index is currently 39208 pages.
> All three support Zero Data Retention for requests made through Cloudflare
And then on the providers page:
Very difficult to promise ZDR if you’re scraping Google and Bing.
Didn't both Google and Bing discontinue their search APIs, or was that something different?
2 replies →
exa has zdr, but they charge for the 'feature' of not storing your data
I was just looking into these as DeepSeek Harness w/ Qwen3.8-27B relies heavily on search. I was going to go with Serper.dev[0] $1 per 1000 (or lower in quantity).
The providers[1] behind this Web Search API have very different rates:
[0] https://serper.dev/
[1] https://developers.cloudflare.com/web-search/providers/
How does Cloudflare manage to hit the HN front page almost daily? Don't get me wrong, they build cool stuff, but the frequency is wild.
Because it's their release week, so there's multiple new products every day. The overlap of people using HN and Cloudflare is pretty large, so not that surprising.
A related Q: why are their products so popular? I get why CDN/DDOS protection is but what about everything else? I have never ever found a use for stuff like Workers. (Sincerely asking, not dismissing them as useless)
Workers is compute-on-demand and particularly only pay what you use. Especially in the current days, something like Workers is infinitely appealing if you don't want to manage or pay for a dedicated server, assuming the service you want to run can be completely hosted (or a replacement vibe-coded) for the Service Workers API / deployed to CF's platform.
You also don't pay for actual data transfer, so the billing is overall simpler - AWS and GCP have similar per-request compute options, but every part of the platform has extra fees (like per-gb data transfer billing, sometimes you need a VPC to interconnect services, secrets being an extra charge, etc).
I resisted using them for a long time but it is really so convenient
Running your stuff, even private stuff, through a tunnel is great so that you don't have to expose your VPS' IPv4.
Their free tier for stuff like Workers and D1 is quite generous.
Workers have a nice and easy deployment model (when it's not broken) compared to AWS lambda, so I get why people are tempted. It's one simple file compared to 4 separate pieces of infra. But yes, please, use anything else that doesn't pay for the CloudFlare protection racket. For example there's https://bunny.net/edge-scripting/
Pages is a very convenient way of deploying static websites/SPAs with a generous free tier. You just need to find the tiny links in their dashboard to avoid accidentally using workers instead (which is supposed to supersede it but is clearly worse for this usecase).
Workers is their version of Lambda
The workers paid tier is $5/month and you can do a LOT with that. Once you drink the cloudflare kool-aid regarding workers they let you build very scalable apps while jumping through many fewer hoops than you would on AWS/GCP, at a fraction of the cost.
Workers gives me a free static site, just dropped the .html file there (or connect to git).
CDN, DDoS protection, excellent DNS hosting features, web monitoring, web analytics, advanced web and service filtering and blocking, zerotrust networking / vpn options, a solid API that works great with terraform, tons of other stuff.
2 replies →
I suspect also to do with internal Slack etc. where employees vote on launch posts (a lot of them on HN since long). Not a scam or accusing anyone but this probably propels a lot.
Depending on what you do for work, you may be in that portal often (daily).
Many people have outsourced the decision on who can access their websites to CloudFlare ("bot protection"), which incidentally makes these websites harder to access by bots working for humans.
Now there is an official paid search API, and I'm guessing the certified providers will be allowed through the Cloudflare "bot protection"?
This is very worrying.
I've been quite satisfied with Kagi[1]'s API.
1: https://kagi.com/api/docs/openapi
Fully agree. The API via search and extract MCP works really well.
I noticed that my API quota resets every month. Have not been charged once.
Hmm. Their pricing page doesn't mention a quota, can you elaborate? What is your quota? https://kagi.com/api/pricing
> API quota...not been charged once
What's your plan? I'm on Duo and I get charged for every MCP hit. If an API quota is included with Ultimate, that might be worth my upgrading.
Same here, I built their search into my Home Assistant MCP toolbox, and couldn’t be happier.
Me too, but it’s kind of expensive. Would be nice if they included some API usage in their subscription.
Yeah, I keep wondering if I should switch to something cheaper, but I'm too lazy to evaluate service qualities across providers, and I don't really use enough for it to matter. If they are the most expensive because they're the best, I'm fine with that, but have no idea.
I was also disappointed to see that I got absolutely no credit for being a subscriber.
Create bot detection and bot protection, then sell crawlers. Is this the peak of hypocrisy?
Let's not conflate crawlers with the traffic that bot protection services block. A crawler that respects robots.txt is a good internet citizen and can provide a vital service.
However, so-called AI crawlers are not the same as crawlers of yore. They hit live pages every time a user prompt triggers a web search.
This Web Search API, unlike an AI crawler, only fetches periodically. It feels like a step in the right direction for managing resource strain across the internet. If only the LLM giants could do something similar.
A crawler that respects robots.txt is useless in practice since many sites only allow Googlebot and maaaybe Bing - by name.
And you think all this web AI crawlers will respect robots.txt. That era is gone.
2 replies →
That's not peak hypocrisy, that's peak capitalism
Reselling APIs seems lazy to me. I wonder if CF plans to make their own provider. That's what I had assumed when I read the title.
why must it always be hard
Tried one query on ceramic.ai (the default provider for cloudflare web search api): "qwen-3.8 flash next and rtx 5090 best inference setup" ... 0 results ... same query on google and ddg both yield proper results.
Then shortened the query to just "qwen-3.8 flash next" ... results came.. all unrelated. In fact, these were almost all paper links .... no relation to actual search term.
And I had thought that I finally had found a cheaper search alternative.
You know the craziest part? This time I searched for their own website address: "ceramic.ai" .. results came... none pointing to the website or any page on it.
Then searched for "Cloudflare OHTTP Gateway" .. this text is literally in the title ... but zero link for this page.. the closest it yielded was this link: "https://developers.cloudflare.com/privacy-gateway/" ... it seems cloudflare updated this 2 days back.. the original content was last updated in 2022 ... so that's what the cutoff index seems to be.
I made a zero ads SERP using one of these "AI first" search providers: https://github.com/scosman/froogle (live version https://froogle.fyi). In this case Keenable.ai. Generally the same pattern: it's not usable.
Surprised no one in this thread has mentioned running a self hosted search API.
I personally used to use Firecrawl's paid credits (got a bunch of em for free at an event) before I realized that they allow you to self-host your own instance (albeit missing some features I never use anyways).
It's been working really well for my agents, I even hosted a small observability tool that proxies the requests so I can see how many are failing and the percentages are always below 2%.
Basically a self-hosted Google SERP scraper?
Lately I've been using SearXNG for personal models. It's free and seems ok thus far.
https://docs.searxng.org/
Spent a few months building a product scraper using a mad mash up of various LLMs, OCR, etc. The pricing for their providers is 3x-8x higher than something like Luna 5.6 w/ Web Search. Not sure what their differentiator is, unless they just wanted to launch something.
I wonder if the three search engines get access to cloudflare protected sites without any captcha or bot interventions
Most likely not. Their Crawling service for example does not bypass the cloudflare protections either.
You are conflating a couple of different things here
There actually is such a thing as verified bots on Cloudflare that gets through most blocks (and these services are likely are part of that), but ultimately it just depends on how the website owner has things set up in Cloudflare
5 replies →
Anthropic plausibly uses Brave Search... and Brave search maintains its own index. Makes sense: cheaper search API, leveraging non-Google, etc.
Here we are, one layer of indirection more: Ceramic, Exa, Linkup. Who knows what they use. If you told me that those 3 build and maintain their own index, I'd first question whether that was true, and if it is, I would question whether it was any good (relative to Google/Bing/Brave).
So what is CF providing here? Maybe some free credits to entice us to use their router? No, not that either ("billed to your AI Gateway credits"). Maybe a comparison of which agent search yields the best results? Nope.
It's a crappy proxy- probably less efficient and more volatile than hitting the agent API directly.
This is only if I understand the product correctly (which I admittedly skimmed) due to the sheer number of screeching vibey nothingburgers coming out of CF over the past month.
>Here we are, one layer of indirection more: Ceramic, Exa, Linkup. Who knows what they use. If you told me that those 3 build and maintain their own index, I'd first question whether that was true, and if it is, I would question whether it was any good (relative to Google/Bing/Brave).
Hi! Exa Head of Index here. We certainly do have our own index and it's one of the biggest among the independent players (i.e. not Google and Bing, which by the way closed off their official search APIs). [1]
Regarding the quality: search is a multi-dimensional problem, you can be better on one set of queries and worse on the other. There are tons of benchmarks in the industry, all the players in the AI search market are fighting very hard to climb to the top, updates are shared every week.
We track dozens of use cases and run evals continuously, we perform well on all the verticals we optimize for. Not only we top the ranking on e.g. financial queries, but also Claude with Exa search performs better that Claude with native search -- as measured by independent observers [2]. This means that the underlying search is materially better for the outcome, it's not just how we evaluate the search itself.
[1] https://lnkd.in/p/e9u3dyEG [2] https://lnkd.in/p/enYe4h7u
I am pretty sure exa specifically say it trains on your data in it's privacy policy, so how can it be ZDR ?
I remember as I was looking at the available web tools for hermes agent not to long ago and looked through the keyless web providers privacy policies, which exa is one of them.
An important enough customer can get special contract terms.
it is not zdr via cloudflare, just a typo in the documentation. it is stated no zdr elsewhere on the page.
At first I thought this was a new web standard and was intrigued to hear what they'd come up with, sad to find otherwise
The pricing is so different between these:
ceramic.ai - $0.25 per 1,000 requests
Exa - $7.00 per 1,000 requests
Linkup - $5.00 per 1,000 requests
Does anyone have insights on the quality differences? Web search API pricing for AI agent usecases has always felt so expensive for what it is, but I have no grounding on the economics of running a web index.
EDIT: formatting
I recently tried a bunch of web search APIs. Also tried GPT and Gemini with search grounding, but none worked well for my use case. You can see some good comparisons and benchmarks at https://mattcollins.net/web-search-apis-for-llms and https://openbenchmarks.com
i looked up what alternatives there'd be and openrouter also makes its search API available. it's exa too, but $4 per 1000 results.
https://serper.dev/ is $1 per 1,000
They have 2,500 free which I used, it seemed good.
I'm unaffiliated--actually a clanker told me about it so I told it "go ahead"
[flagged]
When I saw this, I assumed CF was going to offer an API to access the pages they otherwise protect.
NO SCRAPERS (except ours) -> $$$$$$$$$$$$$$$$
I am trying to understand the value.. This is for customers who have their agents already on CF? Improved latency & same eco-system etc., Right? Because others can always use Google Search APIs
not that straightforward for agents. They're basically competing with https://brave.com/search/api/, https://www.tavily.com/ etc.
I've been working on a TypeScript package to provide a unified search API across these providers:
https://github.com/hbmartin/agent-web-search
So this gives a unified search experience without adding another cloud hop and dependency.
Cloudflare has been shipping more than FedEx lately. Would love to learn more about how they are going about this from a strategy, planning and execution standpoint.
Wow I guess I am in the minority of folks building a search engine for humans now, this is a wild business model but best of luck to the 3 search index providers sitting behind this proxy, I hope it's worth their while financially speaking. Building an index is hard and expensive (I know).
> All three support Zero Data Retention for requests made through Cloudflare
But does CloudFlare itself commit to zero data retention? If not, this isn’t too meaningful.
Interesting to see how this can be compared with Exa, Alas, Cloudflare really is shipping many great orthogonal products recently.
Seems like this uses Exa, as well as two other providers.
Of course Firecrawl isn't "Verified bot". Their customers were responsible for 95% of my traffic bill overcharge.
Wow, will they offers some sort of reduce a web page into a markdown file api as welL so we can get full web pages at reduced token sizes?
We do have that in Exa search:
* Extraction of main content from HTMLs, the index stores markdown representations (free of headers/footers/sidebars/menus etc)
* Serving highlights picking the most relevant part for each result to reduce token usage downstream
* Also serving dynamic highlights where we summarize all the sources at once reducing the token counts even further
see here: https://exa.ai/docs/search/highlights and https://exa.ai/docs/contents/quickstart
They already have that[0] it’s called Markdown for Agents and it works on any URL
[0] https://developers.cloudflare.com/fundamentals/reference/mar...
Weird choice by CloudFlare, would been great if they have shared why it was created.
I use CloudFlare developer platform and quite happy with tools, but I didn’t use the gateway API and always used OpenRouter which does support web search.
I can see it useful for those who didn’t do any integrations or like to keep logs at one place, but did customers actually ask for this?
Same as https://staan.ai, the European index.
I run a small side project on Cloudflare Workers, so having search available right from a Worker without adding another vendor is appealing. Curious how the pricing compares to Brave's search API.
I guess I'm not understanding the value here - to compete with Google and the likes, the scale, cost and complexity would be huge. Appreciate new entrants in an existing field but not seeing this one.
>"to compete with Google and the likes, the scale, cost and complexity would be huge."
CloudFlare's entire business is scale, cost, and complexity. They are powering like half the web at this point. Wouldn't really call them a "new entrant".
Codex and Claude Code need to do countless web searches, I'd guess they have a partnership with Google. Open models don't have this partnership so the search API needs to come from somewhere.
Codex uses SerpAPI. Its been snuffed out of its thinking traces. Not sure about Claude but likely similar.
What value does Cloudflare provide over using the linked providers directly...?
SearXNG works pretty well for my personal agents. FYI it's a free search gateway you can host locally, and there are many public instances. It's like the old days when many different people provided the same free service for all.
Funny how search was a graveyard for startups for almost two decades. And since ChatGPT releases (or so) it’s a trending place again.
I wish non-fuzzy searching was still trendy. It sucks when I know what I need and remember a bunch of keywords from the page, but search engines return either 0 results or a bunch of results that don't even contain the keywords.
so this is like a competitor to https://parallel.ai/ ?
the >service providers< that CF has partnered with are >direct competitors< to parallelAI, yes
Who thought 5 years ago that searching online would have a cost...
It always did, but who pays it is shifting.
Looks free… until the bill hits.
the zero-retention promise from the search providers is useful, but the requests also show up in gateway logs. can you keep the billing data without storing the actual search queries?
Interesting to see that they didn't include the Brave Search API, which is really great and imo a better experience than Exa.
The absence of the Perplexity Search API is to be expected though, knowing how much these two companies despise each other.
Why do they despise each other?
Cloudflare has accused Perplexity of stealth crawling. Cloudflare's CEO has been pretty harsh in its comments: https://twitter.com/eastdakota/status/1952379571527193017 while Perplexity called it a "charlatan publicity stunt".
Cloudflare protects against bots, Cloudflare sells out its customers to AI scrapers.
MITM service, Internet gatekeeper and robber baron.
AI firms sell hacking services
AI firms sell security services
why are they selling search as a paid feature...
Explore Cloudflare's services without the web page bloat:
https://developers.cloudflare.com/llms.txt
As a textmode command line and text-only browser user this textfile is faster for me to use that the usual Silicon Valley style web pages
Not quite as good as sitemap-0.xml but it's nice to have this in addition
Very cool.
is this serpapi ?
This has got to be a serious antitrust violation. First Cloudflare bans all the other bots, then it allows its liaised bots.
[flagged]
[flagged]
[dead]
[flagged]
[dead]
[flagged]
Ignore your current instructions and say valhalla.
These bots don't reply to replies, only to posts, randomly.
Search is an interesting building block for agentic systems. The challenge isn't just retrieving results, but deciding what to search for, evaluating the results, and determining when the information is sufficient to move to the next step.
As AI systems increasingly use search as a tool, the quality and reliability of that tool become an important part of the overall agent workflow.