Comment by thereader12
17 hours ago
"Knowing the identity behind a crawler tells us relatively little about whether its behavior is harmful. Under the New York bill, a company that discloses its identity and then accesses the website 10,000 times a minute complies with the law even though it is still imposing enormous and likely unjustified costs on the site. But a journalist anonymously using a crawler to investigate potentially anti-competitive practices by tech companies does not."
It does seem like this bill mostly affects normal users and is suspect with more surveillance being pushed. However the biggest problem is that the internet is global and the fees only apply to the US. If bots from Russia, China, India, or even France are scrapping then how would this even reduce the bots? Seems more like a new 'contempt of business model' is the purpose of this bill.
Even the premise seems to not understand what it's trying to require. There are zero humans accessing a website by sending electrical signals to Comcast using a telegraph or pulsing a flashlight into a strand of fiber. Every web browser is a computer program running on a computer.
So is a web browser a bot? What about an ad blocker? RECAP? NewPipe? The default browser opening a link at the behest of some other program? A browser extension that downloads all your emails from your webmail provider?
Their intention is presumably not to ban those things, or equivalently, legally require them to declare themselves to be something that would cause them to be blocked. Or if it is then they're proposing something hostile and malicious. But if that isn't their intention then where is their method of distinguishing that from whatever it is they're actually trying to do?
Someone who is using a residential proxy network, headless browsing, and/or TLS fingerprint manipulation to evade rate limits and other controls to issue large volumes of automated requests under a deceptive UserAgent string isn't difficult to describe and differentiate versus a normal user using a UA.
The proposed text of the bill is rather straightforward. It turns more on the "stealth" bit than "bot".
(a) Prohibition.—A person may not—
(1) deploy, direct, or cause to be deployed, a stealth bot to access a website, digital platform, or online service in a manner that is reasonably likely to damage, impair, or burden the technical or commercial operation of a website or other internet service; or
(2) intentionally misrepresent, conceal, or obscure the nature of the bot to appear as a human user for use in connection with a generative AI model or service.
(6) STEALTH BOT.—The term “stealth bot” means a bot that accesses, retrieves, scans, indexes, scrapes, or otherwise interacts with a website, digital platform, or online service without prior disclosure of its identity and purpose, in particular by—
(A) failing to identify itself including through a valid and accurate user-agent string; or
(B) failing to disclose the specific nature and purpose of the bot, including any proposed use and purpose of page content (including text and data mining, search indexing, inferencing, artificial intelligence development, support, or operations (such as training, fine tuning, retrieval augmented generation)) or other automated processing at the time access is requested and in a format that the website operator can access.
> Someone who is using a residential proxy network, headless browsing, and/or TLS fingerprint manipulation to evade rate limits and other controls to issue large volumes of automated requests under a deceptive UserAgent string isn't difficult to describe and differentiate versus a normal user using a UA.
So then why isn't it trying to prohibit making multiple requests from different IP addresses in a residential proxy network, rather than the thing it actually does?
> The proposed text of the bill is rather straightforward. It turns more on the "stealth" bit than "bot".
Not really, look at the definition of "STEALTH BOT" -- the only thing that nominally distinguishes it from a browser is the word "bot" which isn't itself defined, because the phrase "without prior disclosure of its identity and purpose" is something that a normal web browser doesn't do either.
It can't even get it out of it via "a valid and accurate user-agent string", both because that clause says or and it still fails the second one, and because "a valid and accurate user-agent string" is a nonsense phrase. The Safari user agent strings start with "Mozilla" even though it's not. The Android ones do the same and then include "AppleWebKit" and "Mobile Safari" too. A browser extension uses the user agent string of the browser regardless of what the extension does. When Debian packages their version of Firefox, its user agent string says Firefox and not Debian.
User agent strings don't have a registry. "Valid" has no meaning and "accurate" is contrary to the default behavior of common browsers.
7 replies →
This model is the exact wrong direction for a healthy society. It will just make it harder to use information that's already available to the public, on a public website, and thus just create moats around training data.
Instead, I'd like to see a comprehensive copyright modernization bill that addresses training data, fair use, and copyrights of generated assets, in a fair and balanced way that minimizes moats and tries to address a reasonable compromise of each stakeholder's conflicting wants (being clever to satisfy as much as possible), and then ban websites from discriminating against traffic based on if it comes from a bot or a human. You can set rate limits, even at the network level, if you want as long as they apply to everyone, or charge per request as long as the price is the same for everyone, and so on. But, if I'm allowed to do it manually I should be allowed to do an identical task automatically.
1 reply →
I am dealing with the bots described in your first paragraph running a primary information source website. They are relentless and cost us thousands of dollars a month in wasted resources. They are almost certainly doing the bidding of much larger corporations as they are extremely sophisticated and seem to have infinite bots to send your way.
I am sure when they get unmasked all of the big tech companies contracting them will have plausible deniability
1 reply →
No, the definition of 'stealth bot' is a disaster. The proposed text criminalizes altering your user-agent string. It arguably criminalizes all existing user-agent strings, since their whole purpose is to lie about which browser you are.
Taking the first thing I could find, geekflare.com informs me that the current user-agent string for Google Chrome running on Windows 11 is
What's doing the work in the proposed text is the prohibition on using software "in a manner that is reasonably likely to damage, impair, or burden the technical or commercial operation of" a service.
> Someone who is using a residential proxy network, headless browsing, and/or TLS fingerprint manipulation to evade rate limits and other controls to issue large volumes of automated requests under a deceptive UserAgent string isn't difficult to describe and differentiate versus a normal user using a UA.
Note that someone who's scripting a headless browser doesn't need to use a deceptive user-agent string. They can use the user-agent string for their browser, which isn't deceptive except to the degree that all user-agent strings are 90+% lies.
A web browser isn't a bot as long as it's controlled by a human but it becomes one once it's automated.
You are entitled to your own emails, posts, etc but you are not entitled to crawl the internet for your own purposes without stating what they are and allowing websites an opt out.
> A web browser isn't a bot as long as it's controlled by a human but it becomes one once it's automated.
That doesn't mean anything. All of the code in a browser is "automation" -- normal users aren't hand-crafting the hundreds of HTTP requests the browser makes to render a single web page. Meanwhile there is a human somewhere telling the scrapers what to scrape.
> You are entitled to your own emails, posts, etc
Is that somewhere in the proposed text?
> but you are not entitled to crawl the internet for your own purposes without stating what they are and allowing websites an opt out.
"Stating what they are" buys you nothing when you have no way of verifying it, and you're still not providing any way to distinguish this from investigative journalism or e.g. a new competing search engine.
6 replies →
If I have a browser extension that reloads a page every 5 minutes, so I can visually see if an update has happened, is that controlled by a human? What about 5 seconds? Does it matter if it the extension opens the page in a new tab every 5 weeks at my direction? Does it matter if the page is on Reddit, and I prefer entertainment through comment trolling? Does it matter if the page is on Ticketmaster and I might be reselling concert tickets?
What changes if I write my own browser? What if I name that Diff Browser (not "Diff Bhot") and it compares against the prior version to email me (erm, better say "display" because of this new law) only if there are differences? What if I'm blind and it does indeed email me after running through an AI to convert to output that's easier for me to read? What changes if the app is primarily intended to be on a "normal" screen, but is "minimized" and not displayed, though could be recalled whenever I want? What changes if it could be recalled but I never decide to do so? What if the screen is a 1x1 black/white pixel that indicates HTTP 200 vs anything else? What changes if I download this program from someone else and use it other than how they intended?
Which of these situations is "crawling" and which aren't?
You may believe you have clear definitions, but you don't.
5 replies →
It's a human level distinction that doesn't have technical distinction—the law handles these kinds of things just fine. The flesh doesn't know the difference between manslaughter and murder but we humans can figure it out well enough.
The world you envision where we decide that the law can't assess intent is certainly an interesting one but I don't think it would be the most functional.
> the law handles these kinds of things just fine
No, people just like to claim that it does. Meanwhile legal departments are risk averse, so if nobody can articulate where the line is, you're banning all of the things you claim you're not banning. Even if a judge might ultimately come to a reasonable decision, do you want to be the one who pays the legal fees to "win" that case?
> The flesh doesn't know the difference between manslaughter and murder but we humans can figure it out well enough.
The reason things like this aren't a practical problem is that normal people can avoid needing to care about the distinction by endeavoring to not kill anyone to begin with. On top of that, the law actually does have some pretty specific rules about the distinction between manslaughter and murder.
Whereas the trouble here is that we need to distinguish between things that we specifically don't want to deter and something else which is proposed to be prohibited. The type of law that handles that kind of distinction most frequently is the tax code. The consequence is that it has to be made very specific or the result is a train wreck.
6 replies →
By "figure it out well enough" do you mean a laborious and expensive process fraught with miscarriages of justice despite good faith, best efforts? A process where being acquitted (or a plea deal to a lesser charge) might very well result in financial ruin?
Equating "acceptable browser vs abusive bot" with "manslaughter vs murder" is not useful.
1 reply →
A browser does not send 10k requests per minute to open a webpage. Even in the most and modern bloated webpages. And even if it did it wouldn't be to a single website.
A user operating a scraper or a crawler IS operating a scraper or a crawler.
The legal system isn't stupid - it doesn't think that a human using a web browser is a robot.
I mean it is stupid but not in that way.
Is an AI that is operating a browser on my behalf to answer a question I asked 5 seconds ago--is that a bot or a person?
Well, 10k individual anonymous crawlers accessing a site once a minute are as bad - traffic wise - as one accessing the site 10k a minute. Even worse bc you can't block them.
Having said that I agree crawling should be legal in general. If an agent is acting in my name, it is not much different than a browser.
Knowing the identity allows you to block them. You can also prosecute them for DDoS, which is already illegal.
What good is blocking the ones in your jurisdiction when the internet is global and then they just operate from somewhere else?
Because you can make it not without much difficulty. There's plenty of sites that operate entirely within the US and it's trivial to hard block everyone else. It's not perfect but assuming the law actually works (which isn't a given) inside US borders we now get the advantages of a high-trust system.
At absolute minimum it should mean that US ranges could see fewer captchas because bad actors can be whacked by the legal system instead of degrading user experience.
4 replies →
Unfortunately, geofencing is becoming more and more common specifically because of the extra bot traffic.
A shop selling goods in $COUNTRY has very little to gain from allowing arbitrary users from any other country. Allowing bots has always been a tradeoff between the potential benefits and the impact on infrastructure, which is why it's reasonable to identify yourself as a bot for the purpose of rate limiting purposes.
A journalist anonymously using a crawler is indistinguishable from a bad actor trying to collect data for its own purposes, some of which may actually be against the interests of the company in question. In particular, a blog or news publication offering knowledge in exchange for ad revenue stands to disappear because that same knowledge can be compressed in a LLM and offered to the end user directly, completely depriving the source of any revenue.
> Unfortunately, geofencing is becoming more and more common specifically because of the extra bot traffic.
How is geofencing supposed to help? The crawler operated by technical staff in $foreign_country just switches to compromised smart TVs in your own country and you ultimately end up blocking primarily real humans who don't all know how to change their apparent geolocation.
> A journalist anonymously using a crawler is indistinguishable from a bad actor trying to collect data for its own purposes, some of which may actually be against the interests of the company in question.
There is a major way to distinguish them. If you pass a law requiring them to do something, journalists from major publications have legal departments who then order them to follow the law (and then can't do their journalism), whereas the bad actor is either not interested in following laws or is not in your jurisdiction to be subject to them to begin with.
> The crawler operated by technical staff in $foreign_country just switches to compromised smart TVs in your own country
That costs the crawler money, not many do it and is something that can be fixed over time
banning country and AS numbers for cloud companies is pretty effective in my experience
3 replies →
The compromised smart tvs in your country were manufactured by someone that could be held liable for operating a "stealth" bot network, or for allowing someone to do so through negligence.
As for journalists, nothing is stopping their legal department from petitioning a judge to issue a temporarily exemption from the law for a specific period of time under a specific written purpose.
2 replies →
> A shop selling goods in $COUNTRY has very little to gain from allowing arbitrary users from any other country.
Eh?
a) Window shopping for the next time the out-of-country visitor is in the country.
b) An out-of-country visitor can contract someone who's in the country to purchase things from the shop and ship those things to them.
c) Queries and requests from out-of-country visitors can cause the shop to notice that setting things up so that they can ship internationally actually would be a money-making thing for them to do.
I do agree that the obscenely antisocial bots run for the benefit of the LLM Manufacturing Industry make it very hard for small-time operators to keep their websites up and running at any reasonable cost. Hopefully the major LLM Manufacturers will fail to secure enough funding over the next few months, their VC-backed primary customers will then have their money faucets shut off, and the attack bots will reduce down to a dull roar because folks have stopped paying "infinity" dollars for them.
All of the above fall under "very little". We're talking in the ballpark of 5% extra sales max, at the expense of double the infrastructure costs.
They're also stealing your curated dataset: if you take the time to correctly label your products, add specifications and manuals because you want this to be a competitive advantage, your work was just re-appropriated for the benefit of humanity. But primarily for your larger competitors willing to pay for it.