Comment by oasisbob
14 hours ago
Someone who is using a residential proxy network, headless browsing, and/or TLS fingerprint manipulation to evade rate limits and other controls to issue large volumes of automated requests under a deceptive UserAgent string isn't difficult to describe and differentiate versus a normal user using a UA.
The proposed text of the bill is rather straightforward. It turns more on the "stealth" bit than "bot".
(a) Prohibition.—A person may not—
(1) deploy, direct, or cause to be deployed, a stealth bot to access a website, digital platform, or online service in a manner that is reasonably likely to damage, impair, or burden the technical or commercial operation of a website or other internet service; or
(2) intentionally misrepresent, conceal, or obscure the nature of the bot to appear as a human user for use in connection with a generative AI model or service.
(6) STEALTH BOT.—The term “stealth bot” means a bot that accesses, retrieves, scans, indexes, scrapes, or otherwise interacts with a website, digital platform, or online service without prior disclosure of its identity and purpose, in particular by—
(A) failing to identify itself including through a valid and accurate user-agent string; or
(B) failing to disclose the specific nature and purpose of the bot, including any proposed use and purpose of page content (including text and data mining, search indexing, inferencing, artificial intelligence development, support, or operations (such as training, fine tuning, retrieval augmented generation)) or other automated processing at the time access is requested and in a format that the website operator can access.
> Someone who is using a residential proxy network, headless browsing, and/or TLS fingerprint manipulation to evade rate limits and other controls to issue large volumes of automated requests under a deceptive UserAgent string isn't difficult to describe and differentiate versus a normal user using a UA.
So then why isn't it trying to prohibit making multiple requests from different IP addresses in a residential proxy network, rather than the thing it actually does?
> The proposed text of the bill is rather straightforward. It turns more on the "stealth" bit than "bot".
Not really, look at the definition of "STEALTH BOT" -- the only thing that nominally distinguishes it from a browser is the word "bot" which isn't itself defined, because the phrase "without prior disclosure of its identity and purpose" is something that a normal web browser doesn't do either.
It can't even get it out of it via "a valid and accurate user-agent string", both because that clause says or and it still fails the second one, and because "a valid and accurate user-agent string" is a nonsense phrase. The Safari user agent strings start with "Mozilla" even though it's not. The Android ones do the same and then include "AppleWebKit" and "Mobile Safari" too. A browser extension uses the user agent string of the browser regardless of what the extension does. When Debian packages their version of Firefox, its user agent string says Firefox and not Debian.
User agent strings don't have a registry. "Valid" has no meaning and "accurate" is contrary to the default behavior of common browsers.
User agent strings are a sad consequence of websites incorrectly parsing them to enable functionality, resulting in a "every bug becomes a contract" type of situation.
That being said, if you include GOOGLEBOT in your 1k long query string, alongside a very long page describing your crawlers, how to identify them, how to block them and how to set limits, no judge will find you guilty of concealing your purpose.
As for why the law is written as is instead of banning more specific functionality, it's so assholes don't get any ideas on how to bypass it.
I don't think anybody does a decent ua check, webdevs test only in their favorite browser, i.e. Chrome, so only Chrome is allowed. Is it intended? The very purpose of webcompat is to masquerade as Chrome.
How do you even implement an ua check? How many browsers do you know? How many browsers you don't know? How do you implement an ua check for a browser that will appear tomorrow?
I wrote my own http downloader. Do you know about its existence? How do you implement an ua check for it? Can I call it "wget" to make ua check easier for you? Can I call it "Mozilla/5.0" so that it's compatible with your ua check so that you don't need to change anything?
> User agent strings are a sad consequence of websites incorrectly parsing them to enable functionality, resulting in a "every bug becomes a contract" type of situation.
It's a bit old but https://webaim.org/blog/user-agent-string-history/ is always a good parable.
> User agent strings are a sad consequence of websites incorrectly parsing them to enable functionality, resulting in a "every bug becomes a contract" type of situation.
Which is precisely why trying to legislate them is folly. Obviously the reason Android calls itself "Mobile Safari" is so that servers send it the mobile version of the site instead of the desktop one etc.
Which makes "accurate" an unreasonable requirement, since there is already a preexisting non-waivable requirement for inaccurate. When someone is making their "turn a web page into a PDF so you can view it later on the plane" software or whatever, it has to use a user agent string that will cause the site to send the version you need turned into the PDF.
> That being said, if you include GOOGLEBOT in your 1k long query string, alongside a very long page describing your crawlers, how to identify them, how to block them and how to set limits, no judge will find you guilty of concealing your purpose.
First, you're assuming that the person being prosecuted is the entities who are the nominal targets of the bill, when the real problem is when they're not. Because you're proposing a law that says anybody with an inaccurate user agent string is in violation, meanwhile everybody has an inaccurate user agent string.
Moreover, consider what you expect the outcome of legislating something like this is going to be. Sites could then try to either blacklist or whitelist user agents.
If they do the first one, that doesn't work, because the scraping service can change it whenever they want.
If they do the second one, that doesn't work, because now every third party has to start pretending to be a common browser even when they're doing something totally innocent or they get blocked by default for being something else. Which makes the user agent string even less useful than it is now, and subjects innocent parties to arbitrary liability for doing something they have to do or their client is broken.
> As for why the law is written as is instead of banning more specific functionality, it's so assholes don't get any ideas on how to bypass it.
You're describing autocracy. The way you make it so that you always prosecute someone is to make it so that you can always prosecute anyone, which is very bad.
1 reply →
You're slicing the meaning of UA strings too fine.
Bots can easily identify themselves through REP in RFC 9309. This isn't difficult or confusing.
This model is the exact wrong direction for a healthy society. It will just make it harder to use information that's already available to the public, on a public website, and thus just create moats around training data.
Instead, I'd like to see a comprehensive copyright modernization bill that addresses training data, fair use, and copyrights of generated assets, in a fair and balanced way that minimizes moats and tries to address a reasonable compromise of each stakeholder's conflicting wants (being clever to satisfy as much as possible), and then ban websites from discriminating against traffic based on if it comes from a bot or a human. You can set rate limits, even at the network level, if you want as long as they apply to everyone, or charge per request as long as the price is the same for everyone, and so on. But, if I'm allowed to do it manually I should be allowed to do an identical task automatically.
I care a lot less about this issue as a matter of intellectual property than one of capacity consumption and cost.
You have large VC companies evading rate limits and other restrictions and they're causing real economic harm through their actions.
I am dealing with the bots described in your first paragraph running a primary information source website. They are relentless and cost us thousands of dollars a month in wasted resources. They are almost certainly doing the bidding of much larger corporations as they are extremely sophisticated and seem to have infinite bots to send your way.
I am sure when they get unmasked all of the big tech companies contracting them will have plausible deniability
I'm not sure all companies will have deniability. It's pretty easy to see companies which are trying to evade detection or misusing their existing spiders for AI training purposes.
Some examples I've seen have crawlers which mask themselves 99% of the time by faking Chrome, but which will leak their real identity in UA strings which share the overall fingerprint of the swarm.
(I'm looking at you, Replit.)
No, the definition of 'stealth bot' is a disaster. The proposed text criminalizes altering your user-agent string. It arguably criminalizes all existing user-agent strings, since their whole purpose is to lie about which browser you are.
Taking the first thing I could find, geekflare.com informs me that the current user-agent string for Google Chrome running on Windows 11 is
What's doing the work in the proposed text is the prohibition on using software "in a manner that is reasonably likely to damage, impair, or burden the technical or commercial operation of" a service.
> Someone who is using a residential proxy network, headless browsing, and/or TLS fingerprint manipulation to evade rate limits and other controls to issue large volumes of automated requests under a deceptive UserAgent string isn't difficult to describe and differentiate versus a normal user using a UA.
Note that someone who's scripting a headless browser doesn't need to use a deceptive user-agent string. They can use the user-agent string for their browser, which isn't deceptive except to the degree that all user-agent strings are 90+% lies.