Comment by weird-eye-issue
20 hours ago
There is no single answer to this, it entirely depends on the exact service and the exact page
Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections
Meanwhile other services could use a variety of tactics such as simply setting a browser user agent and/or using random data center IPs or residential IPs, etc etc and then the blocking would just come down to whatever the individual page has configured which can actually be pretty hit or miss
From what I've seen, when operating in an environment with bash tools, Claude and ChatGPT will react to a site blocking their normal, attributed fetch tool by calling curl or writing a python script with a different user agent to try and bypass the blocks, rather than respecting the site owners' wishes.
It'd be one thing if they required the user to consciously say "yes, I want you to bypass the block" (still unethical), but the fact that it's just default behavior really says something about the service design alignment.
Claude and ChatGPT are user agents. For obvious reasons, they serve the user, not the websites' owners. (Also, if you don't want people to be able to access some information, then don't put it online.)
My point is that they're not serving the user in that case, since they're taking actions that bypass site terms without consulting the user. And for example, if the site forbids scraping, that attempted block bypass can get the user banned without ever seeing any warnings.
2 replies →
> if you don't want people to be able to access some information
Claude and ChatGPT are not people
4 replies →
Yeah I should have clarified I was primarily referring to the standard web interfaces, however since pretty much all of these tools can execute code these days it is true that technically they can and often will work around blocks. The degree to which this is intentional isn't super clear - if they locked it down too much then it might start refusing to write code that does any sort of web requests which would obviously be an issue for just standard development tasks so it's not super straightforward. But it is true that the built-in tools that they have access to to fetch pages have a hard coded user agent that identifies them
Fair enough yeah, I think the thing that is annoying is that the service design is explicitly disincentivizing keeping an eye on the tasks, with all of the "set it and forget it" features. You really have to put in work to make sure that the model takes breaks to consult you instead of making assumptions (which in OpenAI's internal testing seem to be 'the user wants me to break into a site').
If you hire someone to handle some task for you, do you want them to keep running back to you every time they run into a blocker that they could potentially resolve on their own?
If a developer I hired ignored a GPL license requirement and solved his blocker by using a library without asking me I would be quite concerned.
Nobody would be concerned if they didn’t have access to curl and used wget instead and didn’t ask permission .
However if that solution involved breaking the law yes I absolutely do not want a representative of me breaking the law and yes it is breaking the law whether we like or not under CFAA.
It is not about finding solutions to problems on their own; the solutions should respect the agreements we have made and the law .
1 reply →
That depends if they routinely and cheerfully behave in objectively insane ways.
1 reply →
I'd prefer my little subcontractor to respect the commons on which I depend.
2 replies →
Well yeah, if I ask Claude to look something up on a website so I don’t have to, I don’t give two shits if the website author thinks I shouldn’t be allowed to do that. Just like I don’t care if they think I shouldn’t use Firefox and they block that browser specifically. I’ll route around that however I please and I won’t lose any sleep over it.
Don’t put your shit on the public internet if you’re going to try and micromanage how people consume it.
To be clear, rate limiting to avoid massive usage is different, but that’s not what we’re talking about here.
> I don’t give two shits if the website author thinks…
fair, just don’t cry when website owners don’t “give two shits” about your concerns.
1 reply →
This is not true, Claude and ChatGPT have sent traffic to sites I maintain without any user-agent, it might be they have stopped doing this.
Google has done various versions of this as well, Grok too.
It could be these bots were in some custom harness or something, but they literally sent requests in very weird ways, that humans won't be doing.
I have ended up putting my sites behind vpns or anubis and similar. This is beyond painful.
>Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections
The last part is extremely dubious with some recent reveals.
recent reveals show that OAI and Anthropic give users the ability to go around bot protections or they internally use it?
It’s not exactly recent or a reveal, but I’ve caught at least several OAI models writing Python scripts to access web pages.
5 replies →
I have noticed that agents become extremely creative to fulfill any non refused requests. And then instructions become more of a suggestions.
I've spent a lot of time building datamint.xyz, and dealing with user agents, datacenter IPs, and VPNs gets pretty tricky. What works right now is a fork of the latest Firefox with anti-fingerprinting patches, paired with a high-quality residential proxy network with clean IPs. Even the basic setup needs 11+ code-level patches, and there are still a few edge cases, but this setup should cover 99% of cases.
Thank you for escalating the arms race that's currently destroying the internet. I can't look at the Apple user forums, my health insurance company's portal, pay my county taxes, browse Digikey and Mouser, view Github or whatever forge, etc. etc. without having to jump through Cloudflare or Anubis garbage designed to block your abusive behavior.
I'm just overflowing with joy that you're comfortable wasting my time and money.
I use the automated captcha solvers, stealth browsers, proxy IPs and so on. In regard to ethics, I think this is the right thing to do. The minimum I can do is just to show this is not a matter of excuses or mental gymnastics to trick myself into thinking it's fine.
Even if you said, a lot of the public commons is destroyed, I would NOT use bots and cause more destruction because 'everyone else does it' or 'it's a lost cause'. I say it is the right thing to do because the balance is such that using the internet is actively harmful to the user. That is, using the internet with bots is neutral to the user, and using it without bots, you are being actively harmed and exploited.
If you are using search, this is not a library. You are using your time to train targeted advertising that exploits you; you are selling your search history to the highest bidder. If you are using ChatGPT you are just working as a teacher for free.
On services, you have paid for the product, but the seller is scamming you by not giving you the product. The seller is using dark pattern UI to trick you.
If you are on social media, don't think of this as a public park. You are being treated like a lab animal, your behavior is being studied for the benefit of a corporation.
So the point about the bots is that I think people should be empowered to protect themselves from the internet, which is otherwise actively harmful to their wellbeing.
Even if it's not the GP, if there's profit to be made, somebody will do it. I was reminded of the power of economic motives the other day, when it turned out someone bought an entire trustworthy reseller just so they could compromise the Ledger supply chain by including a tiny eSIM module that OCRed the screen and transmitted the seed.
Incentives are crazy powerful.
2 replies →
"Usage records — which endpoint was called, when, the credit cost, the status, and the target URL."
I would use it, but then the privacy policy says that you would store my search history. I simply want to use the internet in peace without being constantly recorded.
hey,
I didn't get your point, how would you do design our usage records then ?
1 reply →
Here’s what is says on their site: “Courtesy backoff Adaptive backoff keyed to target response codes. When a target returns 429 or 503, we back off immediately, so our fleet never applies denial-of-service pressure to a site.” By the time you get the error you have already crashed the site. Geez.
Anti-fingerprinting is mostly there to be able to use the same browser across many requests, I assume?
I found that a slightly-modified firefox for more stealthy remote-control works fine and gets a pass on all automatic captchas I tested. but I don't do it at scale, I just wanted something to automate a particularly stupid website that I have to use.
Would love to see if my browser is detected -- any recommendations for tests to run?
I have to agree with the (sarcastic) sibling comment. You’re essentially making everyone’s life worse in the name of personal profit.
Please stop. The last thing we need in the world is more greed and selfishness.
Since when have moral appeals worked against the central feature of capitalism?
Only strong regulation can help.
2 replies →