Comment by Doohickey-d
17 hours ago
How well does this (and similar services) deal with anti-bot protection? What happens if I attempt to retrieve a page protected by the cloudflare checkbox or by Anubis? An error, no useful content, or the actual content I want?
> Search API finds web pages for a query and returns structured results. It is powered by DeepSeek's in-house search engine.
Looks like they've built (or are building) their own index and return cached results. What you do with that information is up to you. If you want to visit the live page, you'd need to find a way around bot detection.
One reason (not the only one!) people use Anubis is to prevent the endless flood of requests from AI companies. If Deepseek scrapes once every few hours, that helps the website owner with the traffic problem while allowing access.
If they don't want to allow access, my guess is that deepseek's search crawler respects robots.txt but I don't know.
I have always blamed Chinese AI operators for the hundreds of thousands of residential proxy requests I receive each day, but there's no way to be sure.
They absolutely dwarf real traffic if left unchecked (I blocked all countries outside my target area).
If there’s no way to be sure, then why blame Chinese operators? Because American companies have given you so many reasons to believe they’re acting in good faith?
I don’t particularly trust anyone at this point, and I assume AI traffic on my web servers are all malicious actors, and never bothered to consider what country the malicious bots came from. Mostly because I couldn’t possibly care less, but it kept happening even with geofencing and rate limits, so some American companies who own domestic IP blocks are facilitating them.
1 reply →
Identified AI bots that finally made me cave and put anubis (with very liberal limits) up were anthropic/openai/perplexity according to the headers they sent. They were also over 99% of traffic despite proper robots.txt (site only has a few hundredth visits per day), which they ignore. I don't mind bots or crawlers, but these bots got the same page/query string every time they decided it was time for a crawl, which was about daily, cache control, last modified headers they also never seemed to be interested in. They were very very far from well behaved, unlike search engine crawls, and impossible to tame.
Yeah, but how these residential proxies get in the residential areas? I would guess popular VPN providers are mostly behind this. They sell you VPN and sell bots residential proxy access on the other end.
1 reply →
There is no single answer to this, it entirely depends on the exact service and the exact page
Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections
Meanwhile other services could use a variety of tactics such as simply setting a browser user agent and/or using random data center IPs or residential IPs, etc etc and then the blocking would just come down to whatever the individual page has configured which can actually be pretty hit or miss
From what I've seen, when operating in an environment with bash tools, Claude and ChatGPT will react to a site blocking their normal, attributed fetch tool by calling curl or writing a python script with a different user agent to try and bypass the blocks, rather than respecting the site owners' wishes.
It'd be one thing if they required the user to consciously say "yes, I want you to bypass the block" (still unethical), but the fact that it's just default behavior really says something about the service design alignment.
Claude and ChatGPT are user agents. For obvious reasons, they serve the user, not the websites' owners. (Also, if you don't want people to be able to access some information, then don't put it online.)
8 replies →
Yeah I should have clarified I was primarily referring to the standard web interfaces, however since pretty much all of these tools can execute code these days it is true that technically they can and often will work around blocks. The degree to which this is intentional isn't super clear - if they locked it down too much then it might start refusing to write code that does any sort of web requests which would obviously be an issue for just standard development tasks so it's not super straightforward. But it is true that the built-in tools that they have access to to fetch pages have a hard coded user agent that identifies them
1 reply →
If you hire someone to handle some task for you, do you want them to keep running back to you every time they run into a blocker that they could potentially resolve on their own?
7 replies →
Well yeah, if I ask Claude to look something up on a website so I don’t have to, I don’t give two shits if the website author thinks I shouldn’t be allowed to do that. Just like I don’t care if they think I shouldn’t use Firefox and they block that browser specifically. I’ll route around that however I please and I won’t lose any sleep over it.
Don’t put your shit on the public internet if you’re going to try and micromanage how people consume it.
To be clear, rate limiting to avoid massive usage is different, but that’s not what we’re talking about here.
2 replies →
This is not true, Claude and ChatGPT have sent traffic to sites I maintain without any user-agent, it might be they have stopped doing this.
Google has done various versions of this as well, Grok too.
It could be these bots were in some custom harness or something, but they literally sent requests in very weird ways, that humans won't be doing.
I have ended up putting my sites behind vpns or anubis and similar. This is beyond painful.
>Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections
The last part is extremely dubious with some recent reveals.
recent reveals show that OAI and Anthropic give users the ability to go around bot protections or they internally use it?
7 replies →
I've spent a lot of time building datamint.xyz, and dealing with user agents, datacenter IPs, and VPNs gets pretty tricky. What works right now is a fork of the latest Firefox with anti-fingerprinting patches, paired with a high-quality residential proxy network with clean IPs. Even the basic setup needs 11+ code-level patches, and there are still a few edge cases, but this setup should cover 99% of cases.
Thank you for escalating the arms race that's currently destroying the internet. I can't look at the Apple user forums, my health insurance company's portal, pay my county taxes, browse Digikey and Mouser, view Github or whatever forge, etc. etc. without having to jump through Cloudflare or Anubis garbage designed to block your abusive behavior.
I'm just overflowing with joy that you're comfortable wasting my time and money.
5 replies →
"Usage records — which endpoint was called, when, the credit cost, the status, and the target URL."
I would use it, but then the privacy policy says that you would store my search history. I simply want to use the internet in peace without being constantly recorded.
2 replies →
Here’s what is says on their site: “Courtesy backoff Adaptive backoff keyed to target response codes. When a target returns 429 or 503, we back off immediately, so our fleet never applies denial-of-service pressure to a site.” By the time you get the error you have already crashed the site. Geez.
Anti-fingerprinting is mostly there to be able to use the same browser across many requests, I assume?
I found that a slightly-modified firefox for more stealthy remote-control works fine and gets a pass on all automatic captchas I tested. but I don't do it at scale, I just wanted something to automate a particularly stupid website that I have to use.
Would love to see if my browser is detected -- any recommendations for tests to run?
I have to agree with the (sarcastic) sibling comment. You’re essentially making everyone’s life worse in the name of personal profit.
Please stop. The last thing we need in the world is more greed and selfishness.
3 replies →
usually a combination of something like camoufox and a residential ip address
you can bypass those easily except Google Captcha v3, you need to have your own proxy swarm to make it at scale.