Comment by Havoc
8 days ago
Really feels like this needs some sort of unified possibly legal approach to get these fkers to behave.
Search era clearly proved it is possible to crawl respectfully - the AI crawlers have just decided not to. They need to be disincentivized from doing this
the problem in many cases is that even if such a law is made it likely
- is hard to enforce
- misses bite, i.e. it makes you more money to break it then any penalties
but in general yes, a site which indicates they don't want to be crawled by AI bots but still gets crawled should be handled similar to someone with house ban on a shop forcing them self into the shop
given how severely messed up some millennia cyber security laws are I wonder if crawlers bypassing Anubis could be interpreted as "circumventing digital access controls/protections" or similar, especially given that its done to make copies of copyrighted material ;=)
I really don't get this type of hostility
If you put something in public domain people are going to access it unless you put it behind a paywall but you don't want to do it because that would limit access or people wouldn't pay for it to begin with (ex. your blog nobody wants to pay for)
There's no law against scraping, and we've already past the CFAA argument
Look at it from a lens of harm rather than legality. The hostility comes from people having to pay thousands in bandwidth costs and having services degraded. These AI companies incur huge costs from their wasteful negligence. It's not reasonable.
It’s not quite as simple as “putting something in public domain”. The problem is the server costs to keep that thing in the public domain.
I really don't get why you'd use an ethnic slur as your HN username. Does it make your secret place all tingly when you imagine those who understand, spot it?
The problem here is that some websites, in this case independent open source code forges, can't be put behind paywalls and cannot maintain availability under the load of scrapers.
Where I live, people often put books in boxes out on the sidewalk as giveaways. They're worth nothing, but I love browsing through them if I have time, and sometimes I find something that does interest me. Books I want to throw away I never throw away, I put them out for others to take. I bet most of them just perish in the rain, but it's fine, it's better than throwing them in the trash right away.
It's something people do for people. It's not "in the public domain" for companies to gobble up with machines.
> There's no law against scraping
There's no law against incurring as heavy and pernicious social and material costs to commercial scrapers as is physically possible within legal bounds, either. So what's the problem?