Comment by corbet

3 hours ago

https://lwn.net/Articles/1080822/ Do you really want to be a part of the scraper problem?

From the outside, I don't see the problem. The sites I visit, including LWN, never seem slow or have downtime as a result of this increase in traffic. I hear complaints from people hosting small sites, but they never seem to include concrete examples of downtime or measurably bad user experience. Why does it matter if the server load is high if everything stays functioning? Going from 5-20% to 60-80% load hardly seems like a catastrophe to me when the remaining headroom was not going to be used for anything else. Having your data that's public-enough to be scraped/cited/parodied/ridiculed included in a training set also doesn't seem like a problem. Are they struggling to pay bandwidth usage bills? Is there some actually-necessary intervention required to keep things running smooth, as opposed to panicking and taking unnecessary preventative action?