Comment by AnthonyMouse
16 hours ago
Even the premise seems to not understand what it's trying to require. There are zero humans accessing a website by sending electrical signals to Comcast using a telegraph or pulsing a flashlight into a strand of fiber. Every web browser is a computer program running on a computer.
So is a web browser a bot? What about an ad blocker? RECAP? NewPipe? The default browser opening a link at the behest of some other program? A browser extension that downloads all your emails from your webmail provider?
Their intention is presumably not to ban those things, or equivalently, legally require them to declare themselves to be something that would cause them to be blocked. Or if it is then they're proposing something hostile and malicious. But if that isn't their intention then where is their method of distinguishing that from whatever it is they're actually trying to do?
Someone who is using a residential proxy network, headless browsing, and/or TLS fingerprint manipulation to evade rate limits and other controls to issue large volumes of automated requests under a deceptive UserAgent string isn't difficult to describe and differentiate versus a normal user using a UA.
The proposed text of the bill is rather straightforward. It turns more on the "stealth" bit than "bot".
(a) Prohibition.—A person may not—
(1) deploy, direct, or cause to be deployed, a stealth bot to access a website, digital platform, or online service in a manner that is reasonably likely to damage, impair, or burden the technical or commercial operation of a website or other internet service; or
(2) intentionally misrepresent, conceal, or obscure the nature of the bot to appear as a human user for use in connection with a generative AI model or service.
(6) STEALTH BOT.—The term “stealth bot” means a bot that accesses, retrieves, scans, indexes, scrapes, or otherwise interacts with a website, digital platform, or online service without prior disclosure of its identity and purpose, in particular by—
(A) failing to identify itself including through a valid and accurate user-agent string; or
(B) failing to disclose the specific nature and purpose of the bot, including any proposed use and purpose of page content (including text and data mining, search indexing, inferencing, artificial intelligence development, support, or operations (such as training, fine tuning, retrieval augmented generation)) or other automated processing at the time access is requested and in a format that the website operator can access.
> Someone who is using a residential proxy network, headless browsing, and/or TLS fingerprint manipulation to evade rate limits and other controls to issue large volumes of automated requests under a deceptive UserAgent string isn't difficult to describe and differentiate versus a normal user using a UA.
So then why isn't it trying to prohibit making multiple requests from different IP addresses in a residential proxy network, rather than the thing it actually does?
> The proposed text of the bill is rather straightforward. It turns more on the "stealth" bit than "bot".
Not really, look at the definition of "STEALTH BOT" -- the only thing that nominally distinguishes it from a browser is the word "bot" which isn't itself defined, because the phrase "without prior disclosure of its identity and purpose" is something that a normal web browser doesn't do either.
It can't even get it out of it via "a valid and accurate user-agent string", both because that clause says or and it still fails the second one, and because "a valid and accurate user-agent string" is a nonsense phrase. The Safari user agent strings start with "Mozilla" even though it's not. The Android ones do the same and then include "AppleWebKit" and "Mobile Safari" too. A browser extension uses the user agent string of the browser regardless of what the extension does. When Debian packages their version of Firefox, its user agent string says Firefox and not Debian.
User agent strings don't have a registry. "Valid" has no meaning and "accurate" is contrary to the default behavior of common browsers.
User agent strings are a sad consequence of websites incorrectly parsing them to enable functionality, resulting in a "every bug becomes a contract" type of situation.
That being said, if you include GOOGLEBOT in your 1k long query string, alongside a very long page describing your crawlers, how to identify them, how to block them and how to set limits, no judge will find you guilty of concealing your purpose.
As for why the law is written as is instead of banning more specific functionality, it's so assholes don't get any ideas on how to bypass it.
4 replies →
You're slicing the meaning of UA strings too fine.
Bots can easily identify themselves through REP in RFC 9309. This isn't difficult or confusing.
This model is the exact wrong direction for a healthy society. It will just make it harder to use information that's already available to the public, on a public website, and thus just create moats around training data.
Instead, I'd like to see a comprehensive copyright modernization bill that addresses training data, fair use, and copyrights of generated assets, in a fair and balanced way that minimizes moats and tries to address a reasonable compromise of each stakeholder's conflicting wants (being clever to satisfy as much as possible), and then ban websites from discriminating against traffic based on if it comes from a bot or a human. You can set rate limits, even at the network level, if you want as long as they apply to everyone, or charge per request as long as the price is the same for everyone, and so on. But, if I'm allowed to do it manually I should be allowed to do an identical task automatically.
I care a lot less about this issue as a matter of intellectual property than one of capacity consumption and cost.
You have large VC companies evading rate limits and other restrictions and they're causing real economic harm through their actions.
I am dealing with the bots described in your first paragraph running a primary information source website. They are relentless and cost us thousands of dollars a month in wasted resources. They are almost certainly doing the bidding of much larger corporations as they are extremely sophisticated and seem to have infinite bots to send your way.
I am sure when they get unmasked all of the big tech companies contracting them will have plausible deniability
I'm not sure all companies will have deniability. It's pretty easy to see companies which are trying to evade detection or misusing their existing spiders for AI training purposes.
Some examples I've seen have crawlers which mask themselves 99% of the time by faking Chrome, but which will leak their real identity in UA strings which share the overall fingerprint of the swarm.
(I'm looking at you, Replit.)
No, the definition of 'stealth bot' is a disaster. The proposed text criminalizes altering your user-agent string. It arguably criminalizes all existing user-agent strings, since their whole purpose is to lie about which browser you are.
Taking the first thing I could find, geekflare.com informs me that the current user-agent string for Google Chrome running on Windows 11 is
What's doing the work in the proposed text is the prohibition on using software "in a manner that is reasonably likely to damage, impair, or burden the technical or commercial operation of" a service.
> Someone who is using a residential proxy network, headless browsing, and/or TLS fingerprint manipulation to evade rate limits and other controls to issue large volumes of automated requests under a deceptive UserAgent string isn't difficult to describe and differentiate versus a normal user using a UA.
Note that someone who's scripting a headless browser doesn't need to use a deceptive user-agent string. They can use the user-agent string for their browser, which isn't deceptive except to the degree that all user-agent strings are 90+% lies.
A web browser isn't a bot as long as it's controlled by a human but it becomes one once it's automated.
You are entitled to your own emails, posts, etc but you are not entitled to crawl the internet for your own purposes without stating what they are and allowing websites an opt out.
> A web browser isn't a bot as long as it's controlled by a human but it becomes one once it's automated.
That doesn't mean anything. All of the code in a browser is "automation" -- normal users aren't hand-crafting the hundreds of HTTP requests the browser makes to render a single web page. Meanwhile there is a human somewhere telling the scrapers what to scrape.
> You are entitled to your own emails, posts, etc
Is that somewhere in the proposed text?
> but you are not entitled to crawl the internet for your own purposes without stating what they are and allowing websites an opt out.
"Stating what they are" buys you nothing when you have no way of verifying it, and you're still not providing any way to distinguish this from investigative journalism or e.g. a new competing search engine.
Using a computer to do "by hand" is a vague concept, yes, but laws routinely embrace vague concepts. You used it yourself! "Hand-crafting", you said. It could be pinned down more closely to something like a single action that happens quite soon after you press a button or ... and further wrangling could ensue because it's never going to have all the vagueness removed. It's probably relative to current norms of computer use and what we do or don't automate casually (and thus describe as "by hand").
1 reply →
> All of the code in a browser is "automation"
I don't think you are interpreting what I said in good faith, but here goes:
If you use an API to control a browser into navigating to a certain webpage, as opposed to using human interface devices to assert said control.
And no, AI is not a human interface device.
3 replies →
If I have a browser extension that reloads a page every 5 minutes, so I can visually see if an update has happened, is that controlled by a human? What about 5 seconds? Does it matter if it the extension opens the page in a new tab every 5 weeks at my direction? Does it matter if the page is on Reddit, and I prefer entertainment through comment trolling? Does it matter if the page is on Ticketmaster and I might be reselling concert tickets?
What changes if I write my own browser? What if I name that Diff Browser (not "Diff Bhot") and it compares against the prior version to email me (erm, better say "display" because of this new law) only if there are differences? What if I'm blind and it does indeed email me after running through an AI to convert to output that's easier for me to read? What changes if the app is primarily intended to be on a "normal" screen, but is "minimized" and not displayed, though could be recalled whenever I want? What changes if it could be recalled but I never decide to do so? What if the screen is a 1x1 black/white pixel that indicates HTTP 200 vs anything else? What changes if I download this program from someone else and use it other than how they intended?
Which of these situations is "crawling" and which aren't?
You may believe you have clear definitions, but you don't.
> If I have a browser extension that reloads a page every 5 minutes, so I can visually see if an update has happened, is that controlled by a human?
No, but nobody cares if you reload one page every 5 minutes
> What about 5 seconds?
Definitely no, people will start caring after you do this for an hour or so.
> Does it matter if it the extension opens the page in a new tab every 5 weeks at my direction?
No, if you don't press a button to make it happen every time, it's automated. Also, nobody cares, you are not the intended target here.
> Does it matter if the page is on Reddit, and I prefer entertainment through comment trolling?
No, but being entertained at others' expense suggests some emotional issues you may want to check on.
> Does it matter if the page is on Ticketmaster and I might be reselling concert tickets?
Yes. Scalpers are bad for everyone but themselves.
> What changes if I write my own browser?
Nothing.
> What if I name that Diff Browser (not "Diff Bhot") and it compares against the prior version to email me (erm, better say "display" because of this new law) only if there are differences?
It's still automated, thus a bot. There might be a fair use case if you could've refreshed the pages manually, but what you are describing is RSS.
> What if I'm blind and it does indeed email me after running through an AI to convert to output that's easier for me to read?
I've addressed it here: https://news.ycombinator.com/item?id=50311855
You are unhappy with my response, though I have reason to believe you are biased given your recent loss. I'm sorry for that, though your argument there is the same but with less mass behind the wrecking ball.
> What changes if the app is primarily intended to be on a "normal" screen, but is "minimized" and not displayed, though could be recalled whenever I want?
Are you recalling it manually or automatically?
> What changes if it could be recalled but I never decide to do so?
Then no crawling happens so you don't have to worry about anything?
> What if the screen is a 1x1 black/white pixel that indicates HTTP 200 vs anything else?
This would be automation, since a browser does not behave that way for a normal user.
> What changes if I download this program from someone else and use it other than how they intended?
This is not about programs, it's about websites being crawled against the wishes of their authors.
> You may believe you have clear definitions, but you don't.
Once again, I'm sorry for your loss and any grief my message may have caused you.
4 replies →
It's a human level distinction that doesn't have technical distinction—the law handles these kinds of things just fine. The flesh doesn't know the difference between manslaughter and murder but we humans can figure it out well enough.
The world you envision where we decide that the law can't assess intent is certainly an interesting one but I don't think it would be the most functional.
> the law handles these kinds of things just fine
No, people just like to claim that it does. Meanwhile legal departments are risk averse, so if nobody can articulate where the line is, you're banning all of the things you claim you're not banning. Even if a judge might ultimately come to a reasonable decision, do you want to be the one who pays the legal fees to "win" that case?
> The flesh doesn't know the difference between manslaughter and murder but we humans can figure it out well enough.
The reason things like this aren't a practical problem is that normal people can avoid needing to care about the distinction by endeavoring to not kill anyone to begin with. On top of that, the law actually does have some pretty specific rules about the distinction between manslaughter and murder.
Whereas the trouble here is that we need to distinguish between things that we specifically don't want to deter and something else which is proposed to be prohibited. The type of law that handles that kind of distinction most frequently is the tax code. The consequence is that it has to be made very specific or the result is a train wreck.
Exactly.
> a stealth bot to access a website, digital platform, or online service in a manner that is reasonably likely to damage, impair, or burden the technical or commercial operation of a website or other internet service
"stealth" - what does that mean? This is largely a meaningless term. "damage, impair, or burden" - again, too vague. It would be good to specify something specific, e.g. so many tens of thousand of damage (reciprocal to the assumed fine - a journalist or user is unlikely to be reselling data at this scale, an AI scraper may incur or inflict such costs). "that is reasonably likely to" - this phrase is doing some heavy, heavy lifting, but like others have mentioned, there aren't very many "reasonable" examples, and those have to be defined in a court of law.
Like you and others have said, the immediate reaction would be legal departments essentially banning any and all access, because there would be room for someone to file a suit, and even if you win, there's no guarantee you get your fees back...
1 reply →
Agreed. The law works pretty well for murder/manslaughter because it's a low frequency event that everyone viscerally understands and generally has evolutionarily-ingrained shared values about.
It works less well for other things, like, for example, white collar crime, even when it is low frequency, and even when it is built on financial concepts and norms that have existed at least for decades if not centuries, there's still plenty of gray area for reasonable doubt and spin.
It has pretty much not worked at all for computer and internet regulation except where it could be directly tied back to pre-internet crime definitions (eg. theft of funds). But data privacy, encryption, intellectual property, "hacking", etc, has all been a dumpster fire. And now we pour agentic gasoline on the whole thing? Our legal system is not equipped, least of all not with the current gerontocracy that is beyond clueless about any of this stuff.
> normal people can avoid needing to care about the distinction by endeavoring to not kill anyone to begin with
It's the same thing here, a normal person will never run afoul of this law. It is entirely industry professionals who will be affected and they have legal departments who will figure out where the line is to the millimeter. Sure translating to legalese is hard and I expect this kind of legislation will be mostly common law after a bunch of lawsuits but the giveaway that it will be fine is that assigning "bot/not bot" to a given situation is something we all agree on in the industry. There isn't really that much borderline here. You can't accidentally build a web scraper operating off a residential proxy network and it's pretty obvious when you decide to disguise your traffic because you don't have the consent of site you're scraping.
Even the horribly contrived borderline examples boil down to "but I should have the right to actively deceive and circumvent security measures designed specifically to keep me out" which is a pretty tough sell.
2 replies →
By "figure it out well enough" do you mean a laborious and expensive process fraught with miscarriages of justice despite good faith, best efforts? A process where being acquitted (or a plea deal to a lesser charge) might very well result in financial ruin?
Equating "acceptable browser vs abusive bot" with "manslaughter vs murder" is not useful.
And this is a society where many of the most powerful corporations got started by just blatantly ignoring the laws of their industry. See: Uber, OpenAI, etc.
A browser does not send 10k requests per minute to open a webpage. Even in the most and modern bloated webpages. And even if it did it wouldn't be to a single website.
A user operating a scraper or a crawler IS operating a scraper or a crawler.
The legal system isn't stupid - it doesn't think that a human using a web browser is a robot.
I mean it is stupid but not in that way.
Is an AI that is operating a browser on my behalf to answer a question I asked 5 seconds ago--is that a bot or a person?